March 06, 2020

Outline

  • K Nearest Neighbor algorithm
  • Ensemble learning strategy
  • Bagging: bootstrapping, random forest
  • Boosting: Adaptive Boosting (AdaBoost), Gradient Boosting Machine (GBM) and eXtreme Gradient Boosting (XGBoost)
  • R/Python demo

Overview of K Nearest Neighbor (KNN)

  • Key ideas: data points closer to each other tend to behave similarly
  • Lazy learner: no active model-building is done with training dataset
    • Vs. active learner
    • No training time, long prediction time. Too slow for an online system.
  • Instance-based and distance-based learning
    • Sensitive towards irrelevant attributes
    • Sensitive towards outliers and noisy data
    • Requires data preprocessing: conversion from categorical to numerical attributes, standardization, normalization
  • Non-parametric method: no assumption of data distribution
    • Could detect highly complex nonlinear decision boundary

Variance vs. Bias Tradeoff

  • \(Error = Variance + Bias\)
  • Variance: stability of model and predictions
  • Bias: accuracy of the model/methodology
  • Overfitting: high variance, low bias
  • “No Free Lunch” theorem: trade off between variance and bias, no single algorithm will outperform other algorithms on all datasets

Hyper-Parameters of KNN

  • Number of neighbors (K)
    • Too large: high bias
    • Too small: high variance (overfitting)
  • Distance functions
    • Euclidean
    • Manhattan
    • Minkowski

Demo Dataset: Diabetes

# install.packages("mlbench")
library(caret)
library(mlbench)
data("PimaIndiansDiabetes")
diabetes <- PimaIndiansDiabetes
str(diabetes)
## 'data.frame':    768 obs. of  9 variables:
##  $ pregnant: num  6 1 8 1 0 5 3 10 2 8 ...
##  $ glucose : num  148 85 183 89 137 116 78 115 197 125 ...
##  $ pressure: num  72 66 64 66 40 74 50 0 70 96 ...
##  $ triceps : num  35 29 0 23 35 0 32 0 45 0 ...
##  $ insulin : num  0 0 0 94 168 0 88 0 543 0 ...
##  $ mass    : num  33.6 26.6 23.3 28.1 43.1 25.6 31 35.3 30.5 0 ...
##  $ pedigree: num  0.627 0.351 0.672 0.167 2.288 ...
##  $ age     : num  50 31 32 21 33 30 26 29 53 54 ...
##  $ diabetes: Factor w/ 2 levels "neg","pos": 2 1 2 1 2 1 2 1 2 2 ...
summary(diabetes)
##     pregnant         glucose         pressure         triceps     
##  Min.   : 0.000   Min.   :  0.0   Min.   :  0.00   Min.   : 0.00  
##  1st Qu.: 1.000   1st Qu.: 99.0   1st Qu.: 62.00   1st Qu.: 0.00  
##  Median : 3.000   Median :117.0   Median : 72.00   Median :23.00  
##  Mean   : 3.845   Mean   :120.9   Mean   : 69.11   Mean   :20.54  
##  3rd Qu.: 6.000   3rd Qu.:140.2   3rd Qu.: 80.00   3rd Qu.:32.00  
##  Max.   :17.000   Max.   :199.0   Max.   :122.00   Max.   :99.00  
##     insulin           mass          pedigree           age       
##  Min.   :  0.0   Min.   : 0.00   Min.   :0.0780   Min.   :21.00  
##  1st Qu.:  0.0   1st Qu.:27.30   1st Qu.:0.2437   1st Qu.:24.00  
##  Median : 30.5   Median :32.00   Median :0.3725   Median :29.00  
##  Mean   : 79.8   Mean   :31.99   Mean   :0.4719   Mean   :33.24  
##  3rd Qu.:127.2   3rd Qu.:36.60   3rd Qu.:0.6262   3rd Qu.:41.00  
##  Max.   :846.0   Max.   :67.10   Max.   :2.4200   Max.   :81.00  
##  diabetes 
##  neg:500  
##  pos:268  
##           
##           
##           
## 

Train and Test Baseline KNN Model

library(caret)

set.seed(188)
train_index <- createDataPartition(diabetes$diabetes, p = 0.7, list = FALSE)

diabetes_train <- diabetes[train_index, ]
diabetes_test <- diabetes[-train_index, ]


model_knn <- train(diabetes ~ ., data = diabetes_train, method = "knn")
predict_knn <- predict(model_knn, newdata = diabetes_test)

Measure Model Performance: Hold-Out Method

confusionMatrix(predict_knn, diabetes_test$diabetes)
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction neg pos
##        neg 126  36
##        pos  24  44
##                                           
##                Accuracy : 0.7391          
##                  95% CI : (0.6773, 0.7946)
##     No Information Rate : 0.6522          
##     P-Value [Acc > NIR] : 0.002949        
##                                           
##                   Kappa : 0.4041          
##  Mcnemar's Test P-Value : 0.155580        
##                                           
##             Sensitivity : 0.8400          
##             Specificity : 0.5500          
##          Pos Pred Value : 0.7778          
##          Neg Pred Value : 0.6471          
##              Prevalence : 0.6522          
##          Detection Rate : 0.5478          
##    Detection Prevalence : 0.7043          
##       Balanced Accuracy : 0.6950          
##                                           
##        'Positive' Class : neg             
## 

Measure Model Performance: Bootstrap Method

print(model_knn)
## k-Nearest Neighbors 
## 
## 538 samples
##   8 predictor
##   2 classes: 'neg', 'pos' 
## 
## No pre-processing
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... 
## Resampling results across tuning parameters:
## 
##   k  Accuracy   Kappa    
##   5  0.6901734  0.3082508
##   7  0.6989885  0.3185598
##   9  0.7121062  0.3449934
## 
## Accuracy was used to select the optimal model using the largest value.
## The final value used for the model was k = 9.

Pre-process Data

pre_process <- preProcess(diabetes_train, method = c("scale", "center"))
pre_process
## Created from 538 samples and 9 variables
## 
## Pre-processing:
##   - centered (8)
##   - ignored (1)
##   - scaled (8)
diabetes_train1 <- predict(pre_process, newdata = diabetes_train)
diabetes_test1 <- predict(pre_process, newdata = diabetes_test)
summary(diabetes_train1)
##     pregnant          glucose           pressure           triceps       
##  Min.   :-1.1089   Min.   :-3.7915   Min.   :-3.61879   Min.   :-1.2893  
##  1st Qu.:-0.8243   1st Qu.:-0.6594   1st Qu.:-0.36502   1st Qu.:-1.2893  
##  Median :-0.2550   Median :-0.1270   Median : 0.05482   Median : 0.1813  
##  Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.00000   Mean   : 0.0000  
##  3rd Qu.: 0.5989   3rd Qu.: 0.6169   3rd Qu.: 0.47466   3rd Qu.: 0.7328  
##  Max.   : 3.7299   Max.   : 2.4414   Max.   : 2.78379   Max.   : 4.7771  
##     insulin             mass              pedigree            age         
##  Min.   :-0.6787   Min.   :-4.146198   Min.   :-1.1951   Min.   :-1.0365  
##  1st Qu.:-0.6787   1st Qu.:-0.605945   1st Qu.:-0.7106   1st Qu.:-0.7809  
##  Median :-0.5087   Median : 0.009751   Median :-0.2590   Median :-0.3549  
##  Mean   : 0.0000   Mean   : 0.000000   Mean   : 0.0000   Mean   : 0.0000  
##  3rd Qu.: 0.4158   3rd Qu.: 0.596587   3rd Qu.: 0.4550   3rd Qu.: 0.6675  
##  Max.   : 6.5131   Max.   : 4.460722   Max.   : 5.8088   Max.   : 4.0755  
##  diabetes 
##  neg:350  
##  pos:188  
##           
##           
##           
## 
summary(diabetes_test1)
##     pregnant           glucose            pressure          triceps       
##  Min.   :-1.10892   Min.   :-3.79153   Min.   :-3.6188   Min.   :-1.2893  
##  1st Qu.:-0.82428   1st Qu.:-0.72207   1st Qu.:-0.2601   1st Qu.:-1.2893  
##  Median :-0.25501   Median :-0.07999   Median : 0.1598   Median :-0.1251  
##  Mean   :-0.04834   Mean   :-0.01667   Mean   : 0.0263   Mean   :-0.1032  
##  3rd Qu.: 0.31426   3rd Qu.: 0.55426   3rd Qu.: 0.5796   3rd Qu.: 0.6562  
##  Max.   : 2.59135   Max.   : 2.37871   Max.   : 2.1540   Max.   : 1.7745  
##     insulin               mass            pedigree       
##  Min.   :-0.678722   Min.   :-4.1462   Min.   :-1.17414  
##  1st Qu.:-0.678722   1st Qu.:-0.7791   1st Qu.:-0.68668  
##  Median :-0.300427   Median :-0.1442   Median :-0.39959  
##  Mean   :-0.001154   Mean   :-0.1419   Mean   :-0.05735  
##  3rd Qu.: 0.354151   3rd Qu.: 0.4074   3rd Qu.: 0.36673  
##  Max.   : 4.243363   Max.   : 2.9087   Max.   : 5.41402  
##       age           diabetes 
##  Min.   :-1.03648   neg:150  
##  1st Qu.:-0.78089   pos: 80  
##  Median :-0.31229            
##  Mean   : 0.02147            
##  3rd Qu.: 0.58230            
##  Max.   : 3.13827
names(pre_process)
##  [1] "dim"               "bc"                "yj"               
##  [4] "et"                "invHyperbolicSine" "mean"             
##  [7] "std"               "ranges"            "rotation"         
## [10] "method"            "thresh"            "pcaComp"          
## [13] "numComp"           "ica"               "wildcards"        
## [16] "k"                 "knnSummary"        "bagImp"           
## [19] "median"            "data"              "rangeBounds"
typeof(pre_process)
## [1] "list"
str(pre_process)
## List of 21
##  $ dim              : int [1:2] 538 9
##  $ bc               : NULL
##  $ yj               : NULL
##  $ et               : NULL
##  $ invHyperbolicSine: NULL
##  $ mean             : Named num [1:8] 3.9 121.1 69 21 79.8 ...
##   ..- attr(*, "names")= chr [1:8] "pregnant" "glucose" "pressure" "triceps" ...
##  $ std              : Named num [1:8] 3.51 31.93 19.05 16.32 117.63 ...
##   ..- attr(*, "names")= chr [1:8] "pregnant" "glucose" "pressure" "triceps" ...
##  $ ranges           : NULL
##  $ rotation         : NULL
##  $ method           :List of 3
##   ..$ scale : chr [1:8] "pregnant" "glucose" "pressure" "triceps" ...
##   ..$ center: chr [1:8] "pregnant" "glucose" "pressure" "triceps" ...
##   ..$ ignore: chr "diabetes"
##  $ thresh           : num 0.95
##  $ pcaComp          : NULL
##  $ numComp          : NULL
##  $ ica              : NULL
##  $ wildcards        :List of 2
##   ..$ PCA: chr(0) 
##   ..$ ICA: chr(0) 
##  $ k                : num 5
##  $ knnSummary       :function (x, ...)  
##  $ bagImp           : NULL
##  $ median           : NULL
##  $ data             : NULL
##  $ rangeBounds      : num [1:2] 0 1
##  - attr(*, "class")= chr "preProcess"
pre_process$method$scale[1]
## [1] "pregnant"
set.seed(188)
diabetes1 <- predict(pre_process, diabetes)
summary(diabetes1)
##     pregnant           glucose             pressure        
##  Min.   :-1.10892   Min.   :-3.791531   Min.   :-3.618793  
##  1st Qu.:-0.82428   1st Qu.:-0.690751   1st Qu.:-0.365020  
##  Median :-0.25501   Median :-0.126972   Median : 0.159782  
##  Mean   :-0.01448   Mean   :-0.004992   Mean   : 0.007876  
##  3rd Qu.: 0.59890   3rd Qu.: 0.601241   3rd Qu.: 0.579623  
##  Max.   : 3.72990   Max.   : 2.441351   Max.   : 2.783792  
##     triceps            insulin               mass         
##  Min.   :-1.28931   Min.   :-0.678722   Min.   :-4.14620  
##  1st Qu.:-1.28931   1st Qu.:-0.678722   1st Qu.:-0.64443  
##  Median : 0.12005   Median :-0.419441   Median :-0.04156  
##  Mean   :-0.03091   Mean   :-0.000346   Mean   :-0.04251  
##  3rd Qu.: 0.67153   3rd Qu.: 0.403032   3rd Qu.: 0.54849  
##  Max.   : 4.77705   Max.   : 6.513133   Max.   : 4.46072  
##     pedigree             age            diabetes 
##  Min.   :-1.19508   Min.   :-1.036484   neg:500  
##  1st Qu.:-0.69939   1st Qu.:-0.780886   pos:268  
##  Median :-0.31436   Median :-0.354891            
##  Mean   :-0.01717   Mean   : 0.006429            
##  3rd Qu.: 0.44449   3rd Qu.: 0.667499            
##  Max.   : 5.80877   Max.   : 4.075463
train_index1 <- createDataPartition(diabetes1$diabetes, p = 0.7, list = FALSE)

diabetes_train1 <- diabetes1[train_index1, ]
diabetes_test1 <- diabetes1[-train_index1, ]

New Model Using Standardized Dataset

model_knn1 <- train(diabetes ~ ., data = diabetes_train1, method = "knn")
predict_knn1 <- predict(model_knn1, newdata = diabetes_test1)

confusionMatrix(predict_knn1, diabetes_test1$diabetes, positive = "pos")
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction neg pos
##        neg 128  40
##        pos  22  40
##                                           
##                Accuracy : 0.7304          
##                  95% CI : (0.6682, 0.7866)
##     No Information Rate : 0.6522          
##     P-Value [Acc > NIR] : 0.006878        
##                                           
##                   Kappa : 0.3729          
##  Mcnemar's Test P-Value : 0.030850        
##                                           
##             Sensitivity : 0.5000          
##             Specificity : 0.8533          
##          Pos Pred Value : 0.6452          
##          Neg Pred Value : 0.7619          
##              Prevalence : 0.3478          
##          Detection Rate : 0.1739          
##    Detection Prevalence : 0.2696          
##       Balanced Accuracy : 0.6767          
##                                           
##        'Positive' Class : pos             
## 

Tune the KNN Model

model_knn2 <- train(diabetes ~ ., data = diabetes_train1, method = "knn",
                    tuneGrid = data.frame(k = seq(1, 25)),
                    trControl = trainControl(method = "repeatedcv",
                                             number = 5, repeats = 3))
print(model_knn2)
## k-Nearest Neighbors 
## 
## 538 samples
##   8 predictor
##   2 classes: 'neg', 'pos' 
## 
## No pre-processing
## Resampling: Cross-Validated (5 fold, repeated 3 times) 
## Summary of sample sizes: 430, 430, 431, 431, 430, 431, ... 
## Resampling results across tuning parameters:
## 
##   k   Accuracy   Kappa    
##    1  0.7137764  0.3664780
##    2  0.7038191  0.3375583
##    3  0.7298719  0.3901417
##    4  0.7144283  0.3457516
##    5  0.7360909  0.3933307
##    6  0.7385831  0.3998428
##    7  0.7484943  0.4184038
##    8  0.7503519  0.4257615
##    9  0.7510153  0.4271870
##   10  0.7454540  0.4126475
##   11  0.7466771  0.4145063
##   12  0.7497981  0.4195199
##   13  0.7503750  0.4217115
##   14  0.7472828  0.4130209
##   15  0.7540787  0.4291351
##   16  0.7553133  0.4303757
##   17  0.7602688  0.4408109
##   18  0.7590516  0.4363427
##   19  0.7633783  0.4444193
##   20  0.7639841  0.4466424
##   21  0.7608919  0.4368933
##   22  0.7584285  0.4295055
##   23  0.7652244  0.4461837
##   24  0.7596746  0.4327782
##   25  0.7615265  0.4353147
## 
## Accuracy was used to select the optimal model using the largest value.
## The final value used for the model was k = 23.

Sensitivity Analysis of KNN

plot(model_knn2)

Ensemble Learning Motivations

  • Additive model: combine multiple classifiers together and take consensus
  • Different mechanisms to produce multiple classifiers from the same datasets
    • Bagging (Bootstrap AGGregating): create multiple samples (with replacement) from the original dataset and train multiple classifiers in parallel; reduce variance
    • Boosting: an iterative algorithm that induces base (weak) learners sequentially (e.g. assigns different weights to the data points to focus on those tougher cases): reduce bias
    • Random Forest: train large number of decision stumps (smaller decision trees with less nodes) and use committee majority voting to decide the final classification

Illustratration of Benefit of Ensemble Learning

  • Wisdom of the mass
  • Case study
    • 25 weak learners with 30% error rate
    • What is the error rate of the ensemble classifier
    \[ \begin{equation} \begin{split} &=\sum_{i=13}^{25}C_{i}^{25}P^i(1-P)^{25-i}\\ &=\sum_{i=13}^{25}C_{i}^{25}0.7^i0.3^{25-i}\\ &=0.80 \end{split} \end{equation} \]

Bagging vs. Boosting

Bagging: Bootstrap Resampling Strategy

  • Sampling with replacement
  • 0.632 Rule
    • Roughly \(1/3\) data won’t be included in the bootstrap sample, therefore used for validation purpose
    • \(lim_{n\rightarrow\infty}(1-\frac{1}{n})^n = e^{-1}\approx 0.368\)
    • Simulation
mean(sapply(createResample(1:100, times = 10000), function(x) length(unique(x))))
## [1] 63.3761

Algorithms: Random Forest (1)

  • Produce a distribution of simple ML models on subsets of the original data

  • Combine the distribution into one “aggregated” model

  • Train

    1. Pick \(k\) features from the dataset with \(m\) features (\(k \ll m\), e.g. \(\sqrt{m}\))
    2. Among \(k\) features, pick the one that maximizes information gain (or gain ratio) and use it as the node to further split the data
    3. Repeat step 2 to derive individual decision tree
    4. Repeat step 1 - 3 to generate \(n\) number of trees

Algorithms: Random Forest (2)

  • Predict
    1. Apply each individual tree from the \(n\) random trees to the test data point and derive the outcomes and store the \(n\) predicted outcome
    2. Calculate the weight for each predicted outcome
    3. Consider the outcome with the highest weight as the final classication/prediction for the test data point

Adaptive Boosting (AdaBoosting) Illustration

Gradient Boosting Regressor Algorithm

GBM/XGB

  • Gradient Boosting Machine (GBM)
    • Additive model
    • Boost performance of weak learners to strong learners additively and sequentially
    • Use gradient descent to construct new base learners that minimize loss function
    • Gradient descent is used in the function space, instead of in the parameter space (e.g. backpropagation in neural network)
  • Extreme Gradient Boosting (XGB) improves upon GBM
    • Faster using parallel computing
    • Regularization (hyperparameters such as \(\alpha\), \(\lambda\)), pruning (\(\gamma\)), cross validation enabled, etc.

Analogy of GBM

Visualize GBM

Ensemble Model Hyper-Parameters

  • tree-related parameters: maxdepth/max_depth, minsplit/min_samples_split, minbucket/min_samples_leaf, etc.
  • mtry/max_features: optional integer for number of features to randomly select at each split, such as \(\sqrt{n}\) and \(n\) is the number of features, used for random forest or GBM.
  • ntree/n_estimators: number of classifiers
  • shrinkage/learning_rate (\(\eta\)): GBM/XGB, gradient descent algorithm
  • objective function/loss: classification vs. regression
  • subsample, etc.

Property of the Algorithms: Pros and Cons

  • Pros
    • Perform well on a wide variety of tasks
    • Some ensemble methods such as random forest are robust to overfitting
  • Cons
    • More hyper-parameters to tune
    • Interpretability issue
    • Might be sensitive to outliers/noises and overfitting: boosting
    • Higher computational cost

Bagging: Training

model_bag <- train(diabetes ~ ., data = diabetes_train1, method = "treebag")
model_bag
## Bagged CART 
## 
## 538 samples
##   8 predictor
##   2 classes: 'neg', 'pos' 
## 
## No pre-processing
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... 
## Resampling results:
## 
##   Accuracy   Kappa    
##   0.7369069  0.4135043

Bagging: Prediction and Performance Evaluation

predict_bag <- predict(model_bag, newdata = diabetes_test1)
confusionMatrix(predict_bag, diabetes_test1$diabetes)
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction neg pos
##        neg 128  30
##        pos  22  50
##                                           
##                Accuracy : 0.7739          
##                  95% CI : (0.7143, 0.8263)
##     No Information Rate : 0.6522          
##     P-Value [Acc > NIR] : 4.208e-05       
##                                           
##                   Kappa : 0.4898          
##  Mcnemar's Test P-Value : 0.3317          
##                                           
##             Sensitivity : 0.8533          
##             Specificity : 0.6250          
##          Pos Pred Value : 0.8101          
##          Neg Pred Value : 0.6944          
##              Prevalence : 0.6522          
##          Detection Rate : 0.5565          
##    Detection Prevalence : 0.6870          
##       Balanced Accuracy : 0.7392          
##                                           
##        'Positive' Class : neg             
## 

Random Forest

model_rf <- train(diabetes ~ ., data = diabetes_train1, method = "rf")
model_rf
## Random Forest 
## 
## 538 samples
##   8 predictor
##   2 classes: 'neg', 'pos' 
## 
## No pre-processing
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... 
## Resampling results across tuning parameters:
## 
##   mtry  Accuracy   Kappa    
##   2     0.7624434  0.4640791
##   5     0.7529308  0.4486494
##   8     0.7505543  0.4436233
## 
## Accuracy was used to select the optimal model using the largest value.
## The final value used for the model was mtry = 2.

Gradient Boosting Machine

model_gbm <- train(diabetes ~ ., data = diabetes_train1, method = "gbm")
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.1909             nan     0.1000    0.0159
##      2        1.1682             nan     0.1000    0.0116
##      3        1.1445             nan     0.1000    0.0102
##      4        1.1208             nan     0.1000    0.0095
##      5        1.1006             nan     0.1000    0.0085
##      6        1.0832             nan     0.1000    0.0069
##      7        1.0691             nan     0.1000    0.0065
##      8        1.0581             nan     0.1000    0.0044
##      9        1.0433             nan     0.1000    0.0059
##     10        1.0319             nan     0.1000    0.0049
##     20        0.9391             nan     0.1000    0.0007
##     40        0.8469             nan     0.1000    0.0010
##     60        0.8044             nan     0.1000    0.0000
##     80        0.7735             nan     0.1000    0.0002
##    100        0.7475             nan     0.1000   -0.0010
##    120        0.7305             nan     0.1000   -0.0017
##    140        0.7150             nan     0.1000   -0.0007
##    150        0.7078             nan     0.1000   -0.0008
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.1760             nan     0.1000    0.0209
##      2        1.1369             nan     0.1000    0.0173
##      3        1.1031             nan     0.1000    0.0150
##      4        1.0708             nan     0.1000    0.0139
##      5        1.0449             nan     0.1000    0.0100
##      6        1.0197             nan     0.1000    0.0104
##      7        1.0033             nan     0.1000    0.0057
##      8        0.9841             nan     0.1000    0.0091
##      9        0.9674             nan     0.1000    0.0043
##     10        0.9544             nan     0.1000    0.0034
##     20        0.8558             nan     0.1000    0.0001
##     40        0.7502             nan     0.1000   -0.0001
##     60        0.6847             nan     0.1000    0.0001
##     80        0.6300             nan     0.1000   -0.0009
##    100        0.5886             nan     0.1000   -0.0012
##    120        0.5658             nan     0.1000   -0.0014
##    140        0.5353             nan     0.1000   -0.0010
##    150        0.5226             nan     0.1000   -0.0011
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.1692             nan     0.1000    0.0185
##      2        1.1242             nan     0.1000    0.0208
##      3        1.0760             nan     0.1000    0.0205
##      4        1.0458             nan     0.1000    0.0110
##      5        1.0191             nan     0.1000    0.0109
##      6        0.9942             nan     0.1000    0.0104
##      7        0.9688             nan     0.1000    0.0093
##      8        0.9497             nan     0.1000    0.0041
##      9        0.9314             nan     0.1000    0.0054
##     10        0.9184             nan     0.1000    0.0038
##     20        0.8053             nan     0.1000    0.0011
##     40        0.6789             nan     0.1000   -0.0004
##     60        0.5971             nan     0.1000   -0.0021
##     80        0.5370             nan     0.1000    0.0002
##    100        0.4821             nan     0.1000   -0.0006
##    120        0.4440             nan     0.1000   -0.0010
##    140        0.4109             nan     0.1000   -0.0007
##    150        0.3975             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2359             nan     0.1000    0.0222
##      2        1.1957             nan     0.1000    0.0177
##      3        1.1647             nan     0.1000    0.0140
##      4        1.1401             nan     0.1000    0.0113
##      5        1.1160             nan     0.1000    0.0065
##      6        1.0989             nan     0.1000    0.0078
##      7        1.0755             nan     0.1000    0.0114
##      8        1.0573             nan     0.1000    0.0063
##      9        1.0437             nan     0.1000    0.0038
##     10        1.0260             nan     0.1000    0.0067
##     20        0.9087             nan     0.1000    0.0028
##     40        0.8031             nan     0.1000    0.0008
##     60        0.7492             nan     0.1000   -0.0006
##     80        0.7100             nan     0.1000   -0.0003
##    100        0.6792             nan     0.1000   -0.0012
##    120        0.6601             nan     0.1000   -0.0008
##    140        0.6443             nan     0.1000   -0.0008
##    150        0.6398             nan     0.1000   -0.0012
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2215             nan     0.1000    0.0294
##      2        1.1715             nan     0.1000    0.0262
##      3        1.1339             nan     0.1000    0.0136
##      4        1.1016             nan     0.1000    0.0141
##      5        1.0667             nan     0.1000    0.0185
##      6        1.0395             nan     0.1000    0.0107
##      7        1.0144             nan     0.1000    0.0129
##      8        0.9921             nan     0.1000    0.0086
##      9        0.9728             nan     0.1000    0.0072
##     10        0.9570             nan     0.1000    0.0051
##     20        0.8194             nan     0.1000    0.0042
##     40        0.7056             nan     0.1000    0.0009
##     60        0.6433             nan     0.1000   -0.0015
##     80        0.5938             nan     0.1000   -0.0002
##    100        0.5527             nan     0.1000   -0.0013
##    120        0.5203             nan     0.1000   -0.0009
##    140        0.4957             nan     0.1000   -0.0009
##    150        0.4850             nan     0.1000   -0.0020
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2139             nan     0.1000    0.0295
##      2        1.1519             nan     0.1000    0.0274
##      3        1.0935             nan     0.1000    0.0208
##      4        1.0528             nan     0.1000    0.0174
##      5        1.0234             nan     0.1000    0.0103
##      6        0.9921             nan     0.1000    0.0136
##      7        0.9651             nan     0.1000    0.0103
##      8        0.9441             nan     0.1000    0.0070
##      9        0.9194             nan     0.1000    0.0115
##     10        0.8954             nan     0.1000    0.0075
##     20        0.7526             nan     0.1000    0.0033
##     40        0.6207             nan     0.1000    0.0011
##     60        0.5451             nan     0.1000    0.0003
##     80        0.4917             nan     0.1000   -0.0043
##    100        0.4511             nan     0.1000   -0.0024
##    120        0.4076             nan     0.1000   -0.0013
##    140        0.3758             nan     0.1000   -0.0015
##    150        0.3584             nan     0.1000   -0.0016
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2295             nan     0.1000    0.0194
##      2        1.2032             nan     0.1000    0.0118
##      3        1.1769             nan     0.1000    0.0112
##      4        1.1425             nan     0.1000    0.0150
##      5        1.1137             nan     0.1000    0.0139
##      6        1.0856             nan     0.1000    0.0090
##      7        1.0630             nan     0.1000    0.0103
##      8        1.0417             nan     0.1000    0.0092
##      9        1.0228             nan     0.1000    0.0091
##     10        1.0055             nan     0.1000    0.0081
##     20        0.8881             nan     0.1000    0.0037
##     40        0.7803             nan     0.1000   -0.0001
##     60        0.7268             nan     0.1000   -0.0002
##     80        0.6890             nan     0.1000   -0.0009
##    100        0.6590             nan     0.1000   -0.0006
##    120        0.6364             nan     0.1000   -0.0007
##    140        0.6189             nan     0.1000   -0.0002
##    150        0.6082             nan     0.1000   -0.0012
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2016             nan     0.1000    0.0302
##      2        1.1478             nan     0.1000    0.0240
##      3        1.0980             nan     0.1000    0.0202
##      4        1.0587             nan     0.1000    0.0144
##      5        1.0259             nan     0.1000    0.0135
##      6        0.9998             nan     0.1000    0.0116
##      7        0.9799             nan     0.1000    0.0072
##      8        0.9595             nan     0.1000    0.0071
##      9        0.9356             nan     0.1000    0.0109
##     10        0.9147             nan     0.1000    0.0074
##     20        0.7753             nan     0.1000    0.0028
##     40        0.6645             nan     0.1000   -0.0007
##     60        0.5897             nan     0.1000   -0.0007
##     80        0.5408             nan     0.1000   -0.0007
##    100        0.5110             nan     0.1000   -0.0009
##    120        0.4739             nan     0.1000   -0.0006
##    140        0.4418             nan     0.1000   -0.0008
##    150        0.4293             nan     0.1000   -0.0016
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.1942             nan     0.1000    0.0375
##      2        1.1450             nan     0.1000    0.0224
##      3        1.0959             nan     0.1000    0.0225
##      4        1.0486             nan     0.1000    0.0223
##      5        1.0081             nan     0.1000    0.0137
##      6        0.9791             nan     0.1000    0.0137
##      7        0.9501             nan     0.1000    0.0138
##      8        0.9253             nan     0.1000    0.0103
##      9        0.9043             nan     0.1000    0.0085
##     10        0.8840             nan     0.1000    0.0068
##     20        0.7387             nan     0.1000    0.0028
##     40        0.5932             nan     0.1000    0.0003
##     60        0.5177             nan     0.1000   -0.0003
##     80        0.4564             nan     0.1000   -0.0007
##    100        0.4066             nan     0.1000   -0.0004
##    120        0.3626             nan     0.1000   -0.0018
##    140        0.3256             nan     0.1000   -0.0014
##    150        0.3108             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2422             nan     0.1000    0.0160
##      2        1.2047             nan     0.1000    0.0132
##      3        1.1779             nan     0.1000    0.0119
##      4        1.1578             nan     0.1000    0.0097
##      5        1.1364             nan     0.1000    0.0084
##      6        1.1205             nan     0.1000    0.0057
##      7        1.1006             nan     0.1000    0.0077
##      8        1.0882             nan     0.1000    0.0035
##      9        1.0703             nan     0.1000    0.0083
##     10        1.0561             nan     0.1000    0.0052
##     20        0.9595             nan     0.1000    0.0034
##     40        0.8641             nan     0.1000   -0.0009
##     60        0.8082             nan     0.1000    0.0008
##     80        0.7689             nan     0.1000    0.0010
##    100        0.7421             nan     0.1000   -0.0010
##    120        0.7196             nan     0.1000    0.0001
##    140        0.7042             nan     0.1000   -0.0017
##    150        0.6962             nan     0.1000   -0.0006
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2303             nan     0.1000    0.0218
##      2        1.1853             nan     0.1000    0.0183
##      3        1.1524             nan     0.1000    0.0140
##      4        1.1257             nan     0.1000    0.0125
##      5        1.0938             nan     0.1000    0.0118
##      6        1.0650             nan     0.1000    0.0115
##      7        1.0442             nan     0.1000    0.0069
##      8        1.0231             nan     0.1000    0.0062
##      9        1.0061             nan     0.1000    0.0064
##     10        0.9851             nan     0.1000    0.0079
##     20        0.8589             nan     0.1000    0.0028
##     40        0.7511             nan     0.1000   -0.0001
##     60        0.6871             nan     0.1000   -0.0003
##     80        0.6398             nan     0.1000   -0.0006
##    100        0.5989             nan     0.1000   -0.0007
##    120        0.5616             nan     0.1000   -0.0004
##    140        0.5287             nan     0.1000   -0.0008
##    150        0.5142             nan     0.1000   -0.0006
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2213             nan     0.1000    0.0269
##      2        1.1700             nan     0.1000    0.0206
##      3        1.1263             nan     0.1000    0.0178
##      4        1.0879             nan     0.1000    0.0116
##      5        1.0588             nan     0.1000    0.0070
##      6        1.0334             nan     0.1000    0.0120
##      7        1.0120             nan     0.1000    0.0071
##      8        0.9911             nan     0.1000    0.0072
##      9        0.9713             nan     0.1000    0.0069
##     10        0.9546             nan     0.1000    0.0050
##     20        0.8120             nan     0.1000    0.0023
##     40        0.6884             nan     0.1000    0.0008
##     60        0.6076             nan     0.1000   -0.0008
##     80        0.5422             nan     0.1000    0.0006
##    100        0.4957             nan     0.1000   -0.0031
##    120        0.4550             nan     0.1000   -0.0011
##    140        0.4214             nan     0.1000   -0.0014
##    150        0.4046             nan     0.1000   -0.0011
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2647             nan     0.1000    0.0171
##      2        1.2326             nan     0.1000    0.0175
##      3        1.1997             nan     0.1000    0.0150
##      4        1.1732             nan     0.1000    0.0107
##      5        1.1488             nan     0.1000    0.0095
##      6        1.1308             nan     0.1000    0.0084
##      7        1.1102             nan     0.1000    0.0091
##      8        1.0893             nan     0.1000    0.0092
##      9        1.0735             nan     0.1000    0.0065
##     10        1.0611             nan     0.1000    0.0056
##     20        0.9684             nan     0.1000    0.0027
##     40        0.8603             nan     0.1000   -0.0024
##     60        0.8013             nan     0.1000   -0.0005
##     80        0.7644             nan     0.1000    0.0008
##    100        0.7382             nan     0.1000   -0.0012
##    120        0.7181             nan     0.1000   -0.0014
##    140        0.6957             nan     0.1000   -0.0004
##    150        0.6856             nan     0.1000   -0.0011
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2501             nan     0.1000    0.0277
##      2        1.2040             nan     0.1000    0.0216
##      3        1.1573             nan     0.1000    0.0192
##      4        1.1242             nan     0.1000    0.0138
##      5        1.0948             nan     0.1000    0.0142
##      6        1.0647             nan     0.1000    0.0092
##      7        1.0436             nan     0.1000    0.0098
##      8        1.0241             nan     0.1000    0.0067
##      9        1.0036             nan     0.1000    0.0081
##     10        0.9848             nan     0.1000    0.0060
##     20        0.8607             nan     0.1000    0.0012
##     40        0.7294             nan     0.1000    0.0002
##     60        0.6621             nan     0.1000    0.0005
##     80        0.6066             nan     0.1000   -0.0004
##    100        0.5635             nan     0.1000    0.0009
##    120        0.5279             nan     0.1000   -0.0006
##    140        0.5046             nan     0.1000   -0.0017
##    150        0.4874             nan     0.1000   -0.0001
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2419             nan     0.1000    0.0266
##      2        1.1881             nan     0.1000    0.0210
##      3        1.1426             nan     0.1000    0.0150
##      4        1.1040             nan     0.1000    0.0149
##      5        1.0589             nan     0.1000    0.0190
##      6        1.0270             nan     0.1000    0.0155
##      7        0.9945             nan     0.1000    0.0140
##      8        0.9685             nan     0.1000    0.0073
##      9        0.9444             nan     0.1000    0.0094
##     10        0.9201             nan     0.1000    0.0087
##     20        0.7661             nan     0.1000    0.0050
##     40        0.6384             nan     0.1000   -0.0003
##     60        0.5515             nan     0.1000   -0.0001
##     80        0.4880             nan     0.1000   -0.0019
##    100        0.4421             nan     0.1000   -0.0010
##    120        0.4028             nan     0.1000   -0.0009
##    140        0.3670             nan     0.1000   -0.0012
##    150        0.3526             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2426             nan     0.1000    0.0159
##      2        1.2156             nan     0.1000    0.0112
##      3        1.1849             nan     0.1000    0.0111
##      4        1.1677             nan     0.1000    0.0087
##      5        1.1472             nan     0.1000    0.0085
##      6        1.1272             nan     0.1000    0.0066
##      7        1.1138             nan     0.1000    0.0042
##      8        1.1004             nan     0.1000    0.0029
##      9        1.0823             nan     0.1000    0.0074
##     10        1.0701             nan     0.1000    0.0055
##     20        0.9787             nan     0.1000    0.0016
##     40        0.8770             nan     0.1000    0.0023
##     60        0.8211             nan     0.1000   -0.0009
##     80        0.7885             nan     0.1000   -0.0005
##    100        0.7628             nan     0.1000   -0.0009
##    120        0.7425             nan     0.1000   -0.0010
##    140        0.7259             nan     0.1000   -0.0007
##    150        0.7205             nan     0.1000   -0.0013
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2256             nan     0.1000    0.0177
##      2        1.1876             nan     0.1000    0.0172
##      3        1.1544             nan     0.1000    0.0124
##      4        1.1188             nan     0.1000    0.0178
##      5        1.0903             nan     0.1000    0.0137
##      6        1.0637             nan     0.1000    0.0098
##      7        1.0417             nan     0.1000    0.0053
##      8        1.0168             nan     0.1000    0.0112
##      9        0.9955             nan     0.1000    0.0094
##     10        0.9749             nan     0.1000    0.0081
##     20        0.8544             nan     0.1000   -0.0005
##     40        0.7462             nan     0.1000   -0.0003
##     60        0.6803             nan     0.1000   -0.0008
##     80        0.6228             nan     0.1000   -0.0010
##    100        0.5805             nan     0.1000   -0.0008
##    120        0.5480             nan     0.1000   -0.0023
##    140        0.5161             nan     0.1000   -0.0016
##    150        0.4992             nan     0.1000    0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2150             nan     0.1000    0.0253
##      2        1.1677             nan     0.1000    0.0183
##      3        1.1303             nan     0.1000    0.0153
##      4        1.0918             nan     0.1000    0.0135
##      5        1.0555             nan     0.1000    0.0142
##      6        1.0273             nan     0.1000    0.0110
##      7        0.9964             nan     0.1000    0.0136
##      8        0.9717             nan     0.1000    0.0072
##      9        0.9522             nan     0.1000    0.0063
##     10        0.9319             nan     0.1000    0.0063
##     20        0.7966             nan     0.1000   -0.0002
##     40        0.6607             nan     0.1000    0.0007
##     60        0.5839             nan     0.1000   -0.0005
##     80        0.5225             nan     0.1000   -0.0007
##    100        0.4749             nan     0.1000   -0.0013
##    120        0.4296             nan     0.1000   -0.0010
##    140        0.3934             nan     0.1000   -0.0010
##    150        0.3767             nan     0.1000   -0.0006
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2445             nan     0.1000    0.0173
##      2        1.2177             nan     0.1000    0.0071
##      3        1.1865             nan     0.1000    0.0161
##      4        1.1538             nan     0.1000    0.0125
##      5        1.1322             nan     0.1000    0.0105
##      6        1.1074             nan     0.1000    0.0099
##      7        1.0867             nan     0.1000    0.0084
##      8        1.0659             nan     0.1000    0.0092
##      9        1.0483             nan     0.1000    0.0073
##     10        1.0335             nan     0.1000    0.0072
##     20        0.9182             nan     0.1000    0.0029
##     40        0.8013             nan     0.1000    0.0013
##     60        0.7481             nan     0.1000    0.0006
##     80        0.7088             nan     0.1000   -0.0000
##    100        0.6774             nan     0.1000   -0.0008
##    120        0.6544             nan     0.1000   -0.0006
##    140        0.6363             nan     0.1000   -0.0020
##    150        0.6275             nan     0.1000   -0.0023
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2286             nan     0.1000    0.0297
##      2        1.1786             nan     0.1000    0.0222
##      3        1.1339             nan     0.1000    0.0168
##      4        1.0946             nan     0.1000    0.0200
##      5        1.0567             nan     0.1000    0.0196
##      6        1.0227             nan     0.1000    0.0149
##      7        0.9922             nan     0.1000    0.0124
##      8        0.9712             nan     0.1000    0.0082
##      9        0.9547             nan     0.1000    0.0051
##     10        0.9343             nan     0.1000    0.0093
##     20        0.8142             nan     0.1000    0.0009
##     40        0.6890             nan     0.1000   -0.0005
##     60        0.6181             nan     0.1000   -0.0013
##     80        0.5729             nan     0.1000   -0.0008
##    100        0.5372             nan     0.1000   -0.0009
##    120        0.5093             nan     0.1000   -0.0013
##    140        0.4814             nan     0.1000    0.0001
##    150        0.4668             nan     0.1000    0.0002
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2185             nan     0.1000    0.0305
##      2        1.1590             nan     0.1000    0.0279
##      3        1.1026             nan     0.1000    0.0250
##      4        1.0653             nan     0.1000    0.0157
##      5        1.0251             nan     0.1000    0.0168
##      6        0.9930             nan     0.1000    0.0126
##      7        0.9574             nan     0.1000    0.0138
##      8        0.9349             nan     0.1000    0.0090
##      9        0.9123             nan     0.1000    0.0079
##     10        0.8938             nan     0.1000    0.0071
##     20        0.7556             nan     0.1000    0.0018
##     40        0.6285             nan     0.1000    0.0017
##     60        0.5437             nan     0.1000    0.0008
##     80        0.4897             nan     0.1000   -0.0027
##    100        0.4432             nan     0.1000   -0.0015
##    120        0.4012             nan     0.1000   -0.0013
##    140        0.3691             nan     0.1000   -0.0006
##    150        0.3552             nan     0.1000   -0.0014
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2612             nan     0.1000    0.0184
##      2        1.2347             nan     0.1000    0.0121
##      3        1.2035             nan     0.1000    0.0078
##      4        1.1785             nan     0.1000    0.0123
##      5        1.1626             nan     0.1000    0.0044
##      6        1.1420             nan     0.1000    0.0084
##      7        1.1232             nan     0.1000    0.0087
##      8        1.1093             nan     0.1000    0.0059
##      9        1.0915             nan     0.1000    0.0076
##     10        1.0766             nan     0.1000    0.0073
##     20        0.9662             nan     0.1000    0.0001
##     40        0.8561             nan     0.1000   -0.0005
##     60        0.7953             nan     0.1000    0.0002
##     80        0.7552             nan     0.1000    0.0001
##    100        0.7201             nan     0.1000   -0.0010
##    120        0.6975             nan     0.1000   -0.0009
##    140        0.6728             nan     0.1000   -0.0005
##    150        0.6621             nan     0.1000   -0.0016
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2397             nan     0.1000    0.0241
##      2        1.2075             nan     0.1000    0.0117
##      3        1.1740             nan     0.1000    0.0163
##      4        1.1376             nan     0.1000    0.0157
##      5        1.1093             nan     0.1000    0.0130
##      6        1.0743             nan     0.1000    0.0102
##      7        1.0505             nan     0.1000    0.0074
##      8        1.0308             nan     0.1000    0.0078
##      9        1.0105             nan     0.1000    0.0084
##     10        0.9916             nan     0.1000    0.0082
##     20        0.8664             nan     0.1000    0.0027
##     40        0.7387             nan     0.1000    0.0000
##     60        0.6663             nan     0.1000   -0.0009
##     80        0.6160             nan     0.1000   -0.0030
##    100        0.5774             nan     0.1000   -0.0023
##    120        0.5378             nan     0.1000   -0.0026
##    140        0.5025             nan     0.1000   -0.0003
##    150        0.4912             nan     0.1000   -0.0003
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2426             nan     0.1000    0.0220
##      2        1.1857             nan     0.1000    0.0250
##      3        1.1413             nan     0.1000    0.0169
##      4        1.1045             nan     0.1000    0.0165
##      5        1.0682             nan     0.1000    0.0137
##      6        1.0411             nan     0.1000    0.0109
##      7        1.0157             nan     0.1000    0.0078
##      8        0.9878             nan     0.1000    0.0112
##      9        0.9671             nan     0.1000    0.0064
##     10        0.9459             nan     0.1000    0.0083
##     20        0.8030             nan     0.1000    0.0036
##     40        0.6457             nan     0.1000   -0.0000
##     60        0.5648             nan     0.1000   -0.0014
##     80        0.4995             nan     0.1000   -0.0005
##    100        0.4495             nan     0.1000   -0.0004
##    120        0.4053             nan     0.1000   -0.0011
##    140        0.3665             nan     0.1000   -0.0012
##    150        0.3471             nan     0.1000   -0.0001
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2772             nan     0.1000    0.0142
##      2        1.2456             nan     0.1000    0.0116
##      3        1.2244             nan     0.1000    0.0092
##      4        1.2011             nan     0.1000    0.0111
##      5        1.1845             nan     0.1000    0.0052
##      6        1.1645             nan     0.1000    0.0096
##      7        1.1459             nan     0.1000    0.0067
##      8        1.1291             nan     0.1000    0.0062
##      9        1.1147             nan     0.1000    0.0062
##     10        1.1002             nan     0.1000    0.0059
##     20        1.0054             nan     0.1000   -0.0006
##     40        0.9102             nan     0.1000    0.0017
##     60        0.8595             nan     0.1000    0.0004
##     80        0.8295             nan     0.1000    0.0010
##    100        0.8106             nan     0.1000   -0.0004
##    120        0.7904             nan     0.1000   -0.0008
##    140        0.7737             nan     0.1000   -0.0012
##    150        0.7669             nan     0.1000   -0.0006
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2695             nan     0.1000    0.0204
##      2        1.2261             nan     0.1000    0.0200
##      3        1.1854             nan     0.1000    0.0161
##      4        1.1534             nan     0.1000    0.0139
##      5        1.1271             nan     0.1000    0.0088
##      6        1.1076             nan     0.1000    0.0066
##      7        1.0899             nan     0.1000    0.0045
##      8        1.0657             nan     0.1000    0.0105
##      9        1.0486             nan     0.1000    0.0082
##     10        1.0318             nan     0.1000    0.0072
##     20        0.9229             nan     0.1000    0.0019
##     40        0.8237             nan     0.1000   -0.0022
##     60        0.7566             nan     0.1000   -0.0007
##     80        0.7071             nan     0.1000   -0.0023
##    100        0.6692             nan     0.1000   -0.0018
##    120        0.6311             nan     0.1000   -0.0001
##    140        0.5977             nan     0.1000   -0.0024
##    150        0.5794             nan     0.1000    0.0005
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2537             nan     0.1000    0.0266
##      2        1.2056             nan     0.1000    0.0204
##      3        1.1622             nan     0.1000    0.0215
##      4        1.1194             nan     0.1000    0.0202
##      5        1.0871             nan     0.1000    0.0149
##      6        1.0556             nan     0.1000    0.0114
##      7        1.0244             nan     0.1000    0.0134
##      8        0.9970             nan     0.1000    0.0113
##      9        0.9765             nan     0.1000    0.0047
##     10        0.9576             nan     0.1000    0.0053
##     20        0.8433             nan     0.1000    0.0018
##     40        0.7328             nan     0.1000   -0.0010
##     60        0.6583             nan     0.1000   -0.0017
##     80        0.6013             nan     0.1000    0.0010
##    100        0.5508             nan     0.1000   -0.0018
##    120        0.5010             nan     0.1000   -0.0008
##    140        0.4629             nan     0.1000   -0.0009
##    150        0.4443             nan     0.1000   -0.0005
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2372             nan     0.1000    0.0127
##      2        1.1988             nan     0.1000    0.0161
##      3        1.1822             nan     0.1000    0.0056
##      4        1.1597             nan     0.1000    0.0101
##      5        1.1365             nan     0.1000    0.0090
##      6        1.1139             nan     0.1000    0.0099
##      7        1.0954             nan     0.1000    0.0083
##      8        1.0814             nan     0.1000    0.0063
##      9        1.0643             nan     0.1000    0.0072
##     10        1.0480             nan     0.1000    0.0043
##     20        0.9543             nan     0.1000    0.0011
##     40        0.8556             nan     0.1000   -0.0001
##     60        0.7939             nan     0.1000    0.0004
##     80        0.7555             nan     0.1000   -0.0001
##    100        0.7211             nan     0.1000   -0.0008
##    120        0.6914             nan     0.1000   -0.0008
##    140        0.6689             nan     0.1000   -0.0003
##    150        0.6571             nan     0.1000   -0.0007
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2231             nan     0.1000    0.0191
##      2        1.1876             nan     0.1000    0.0153
##      3        1.1442             nan     0.1000    0.0152
##      4        1.1119             nan     0.1000    0.0103
##      5        1.0827             nan     0.1000    0.0120
##      6        1.0570             nan     0.1000    0.0134
##      7        1.0381             nan     0.1000    0.0050
##      8        1.0186             nan     0.1000    0.0058
##      9        1.0017             nan     0.1000    0.0036
##     10        0.9765             nan     0.1000    0.0092
##     20        0.8584             nan     0.1000    0.0028
##     40        0.7313             nan     0.1000    0.0000
##     60        0.6553             nan     0.1000   -0.0005
##     80        0.5916             nan     0.1000   -0.0005
##    100        0.5386             nan     0.1000   -0.0005
##    120        0.4996             nan     0.1000   -0.0008
##    140        0.4661             nan     0.1000   -0.0004
##    150        0.4527             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2079             nan     0.1000    0.0242
##      2        1.1598             nan     0.1000    0.0183
##      3        1.1219             nan     0.1000    0.0162
##      4        1.0920             nan     0.1000    0.0102
##      5        1.0600             nan     0.1000    0.0120
##      6        1.0326             nan     0.1000    0.0109
##      7        1.0044             nan     0.1000    0.0116
##      8        0.9723             nan     0.1000    0.0134
##      9        0.9502             nan     0.1000    0.0089
##     10        0.9351             nan     0.1000    0.0032
##     20        0.8016             nan     0.1000    0.0049
##     40        0.6512             nan     0.1000    0.0019
##     60        0.5544             nan     0.1000   -0.0004
##     80        0.4896             nan     0.1000   -0.0010
##    100        0.4373             nan     0.1000   -0.0008
##    120        0.3888             nan     0.1000   -0.0020
##    140        0.3493             nan     0.1000   -0.0013
##    150        0.3305             nan     0.1000   -0.0011
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2285             nan     0.1000    0.0190
##      2        1.1964             nan     0.1000    0.0161
##      3        1.1673             nan     0.1000    0.0136
##      4        1.1409             nan     0.1000    0.0106
##      5        1.1218             nan     0.1000    0.0072
##      6        1.1024             nan     0.1000    0.0085
##      7        1.0863             nan     0.1000    0.0071
##      8        1.0711             nan     0.1000    0.0079
##      9        1.0573             nan     0.1000    0.0062
##     10        1.0449             nan     0.1000    0.0047
##     20        0.9346             nan     0.1000    0.0028
##     40        0.8340             nan     0.1000    0.0003
##     60        0.7797             nan     0.1000    0.0000
##     80        0.7431             nan     0.1000   -0.0005
##    100        0.7134             nan     0.1000   -0.0007
##    120        0.6940             nan     0.1000   -0.0011
##    140        0.6739             nan     0.1000   -0.0008
##    150        0.6637             nan     0.1000   -0.0008
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2197             nan     0.1000    0.0216
##      2        1.1782             nan     0.1000    0.0177
##      3        1.1399             nan     0.1000    0.0165
##      4        1.1063             nan     0.1000    0.0160
##      5        1.0809             nan     0.1000    0.0129
##      6        1.0537             nan     0.1000    0.0081
##      7        1.0289             nan     0.1000    0.0093
##      8        1.0023             nan     0.1000    0.0099
##      9        0.9815             nan     0.1000    0.0083
##     10        0.9614             nan     0.1000    0.0074
##     20        0.8455             nan     0.1000    0.0010
##     40        0.7327             nan     0.1000   -0.0009
##     60        0.6607             nan     0.1000   -0.0025
##     80        0.6119             nan     0.1000   -0.0011
##    100        0.5654             nan     0.1000   -0.0012
##    120        0.5313             nan     0.1000   -0.0005
##    140        0.4998             nan     0.1000   -0.0004
##    150        0.4833             nan     0.1000   -0.0005
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2099             nan     0.1000    0.0271
##      2        1.1591             nan     0.1000    0.0219
##      3        1.1198             nan     0.1000    0.0135
##      4        1.0789             nan     0.1000    0.0184
##      5        1.0447             nan     0.1000    0.0151
##      6        1.0146             nan     0.1000    0.0124
##      7        0.9852             nan     0.1000    0.0110
##      8        0.9612             nan     0.1000    0.0093
##      9        0.9362             nan     0.1000    0.0087
##     10        0.9123             nan     0.1000    0.0106
##     20        0.7760             nan     0.1000    0.0020
##     40        0.6340             nan     0.1000   -0.0011
##     60        0.5591             nan     0.1000    0.0003
##     80        0.4948             nan     0.1000   -0.0006
##    100        0.4450             nan     0.1000   -0.0008
##    120        0.3969             nan     0.1000    0.0001
##    140        0.3590             nan     0.1000   -0.0012
##    150        0.3440             nan     0.1000   -0.0015
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2572             nan     0.1000    0.0205
##      2        1.2236             nan     0.1000    0.0182
##      3        1.1946             nan     0.1000    0.0126
##      4        1.1699             nan     0.1000    0.0119
##      5        1.1464             nan     0.1000    0.0099
##      6        1.1245             nan     0.1000    0.0104
##      7        1.1064             nan     0.1000    0.0058
##      8        1.0851             nan     0.1000    0.0082
##      9        1.0655             nan     0.1000    0.0071
##     10        1.0487             nan     0.1000    0.0075
##     20        0.9294             nan     0.1000    0.0029
##     40        0.8175             nan     0.1000    0.0008
##     60        0.7573             nan     0.1000    0.0013
##     80        0.7139             nan     0.1000   -0.0007
##    100        0.6797             nan     0.1000   -0.0011
##    120        0.6592             nan     0.1000   -0.0005
##    140        0.6379             nan     0.1000   -0.0009
##    150        0.6293             nan     0.1000   -0.0005
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2353             nan     0.1000    0.0269
##      2        1.1910             nan     0.1000    0.0197
##      3        1.1515             nan     0.1000    0.0185
##      4        1.1075             nan     0.1000    0.0206
##      5        1.0747             nan     0.1000    0.0117
##      6        1.0407             nan     0.1000    0.0149
##      7        1.0163             nan     0.1000    0.0126
##      8        0.9927             nan     0.1000    0.0101
##      9        0.9689             nan     0.1000    0.0095
##     10        0.9505             nan     0.1000    0.0083
##     20        0.8169             nan     0.1000    0.0035
##     40        0.7003             nan     0.1000   -0.0007
##     60        0.6278             nan     0.1000   -0.0009
##     80        0.5695             nan     0.1000    0.0001
##    100        0.5275             nan     0.1000   -0.0001
##    120        0.4873             nan     0.1000   -0.0006
##    140        0.4595             nan     0.1000   -0.0010
##    150        0.4463             nan     0.1000   -0.0008
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2277             nan     0.1000    0.0356
##      2        1.1654             nan     0.1000    0.0264
##      3        1.1161             nan     0.1000    0.0246
##      4        1.0778             nan     0.1000    0.0199
##      5        1.0375             nan     0.1000    0.0164
##      6        1.0066             nan     0.1000    0.0122
##      7        0.9814             nan     0.1000    0.0113
##      8        0.9538             nan     0.1000    0.0102
##      9        0.9318             nan     0.1000    0.0093
##     10        0.9146             nan     0.1000    0.0036
##     20        0.7552             nan     0.1000    0.0017
##     40        0.6225             nan     0.1000    0.0000
##     60        0.5399             nan     0.1000   -0.0007
##     80        0.4706             nan     0.1000   -0.0017
##    100        0.4213             nan     0.1000   -0.0008
##    120        0.3741             nan     0.1000   -0.0011
##    140        0.3373             nan     0.1000   -0.0010
##    150        0.3219             nan     0.1000   -0.0006
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.3100             nan     0.1000    0.0091
##      2        1.2842             nan     0.1000    0.0121
##      3        1.2561             nan     0.1000    0.0087
##      4        1.2326             nan     0.1000    0.0104
##      5        1.2126             nan     0.1000    0.0082
##      6        1.1954             nan     0.1000    0.0071
##      7        1.1749             nan     0.1000    0.0064
##      8        1.1598             nan     0.1000    0.0065
##      9        1.1440             nan     0.1000    0.0065
##     10        1.1285             nan     0.1000    0.0062
##     20        1.0324             nan     0.1000    0.0041
##     40        0.9291             nan     0.1000   -0.0005
##     60        0.8757             nan     0.1000   -0.0001
##     80        0.8416             nan     0.1000   -0.0002
##    100        0.8113             nan     0.1000   -0.0006
##    120        0.7882             nan     0.1000   -0.0004
##    140        0.7666             nan     0.1000   -0.0010
##    150        0.7578             nan     0.1000   -0.0002
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.3029             nan     0.1000    0.0094
##      2        1.2649             nan     0.1000    0.0185
##      3        1.2261             nan     0.1000    0.0158
##      4        1.1972             nan     0.1000    0.0117
##      5        1.1796             nan     0.1000    0.0056
##      6        1.1502             nan     0.1000    0.0123
##      7        1.1287             nan     0.1000    0.0085
##      8        1.1106             nan     0.1000    0.0038
##      9        1.0901             nan     0.1000    0.0086
##     10        1.0741             nan     0.1000    0.0026
##     20        0.9387             nan     0.1000    0.0041
##     40        0.8249             nan     0.1000   -0.0004
##     60        0.7541             nan     0.1000   -0.0011
##     80        0.7034             nan     0.1000   -0.0023
##    100        0.6565             nan     0.1000   -0.0011
##    120        0.6216             nan     0.1000   -0.0003
##    140        0.5917             nan     0.1000   -0.0007
##    150        0.5766             nan     0.1000   -0.0019
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2832             nan     0.1000    0.0220
##      2        1.2263             nan     0.1000    0.0238
##      3        1.1878             nan     0.1000    0.0195
##      4        1.1495             nan     0.1000    0.0151
##      5        1.1210             nan     0.1000    0.0102
##      6        1.0899             nan     0.1000    0.0132
##      7        1.0625             nan     0.1000    0.0109
##      8        1.0348             nan     0.1000    0.0122
##      9        1.0103             nan     0.1000    0.0104
##     10        0.9904             nan     0.1000    0.0093
##     20        0.8548             nan     0.1000    0.0006
##     40        0.7289             nan     0.1000    0.0008
##     60        0.6468             nan     0.1000   -0.0007
##     80        0.5817             nan     0.1000   -0.0002
##    100        0.5282             nan     0.1000   -0.0016
##    120        0.4887             nan     0.1000    0.0001
##    140        0.4547             nan     0.1000   -0.0014
##    150        0.4359             nan     0.1000   -0.0010
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2579             nan     0.1000    0.0214
##      2        1.2233             nan     0.1000    0.0163
##      3        1.1886             nan     0.1000    0.0136
##      4        1.1527             nan     0.1000    0.0112
##      5        1.1305             nan     0.1000    0.0113
##      6        1.1102             nan     0.1000    0.0093
##      7        1.0901             nan     0.1000    0.0091
##      8        1.0737             nan     0.1000    0.0059
##      9        1.0569             nan     0.1000    0.0071
##     10        1.0423             nan     0.1000    0.0055
##     20        0.9276             nan     0.1000    0.0055
##     40        0.8200             nan     0.1000    0.0006
##     60        0.7622             nan     0.1000   -0.0001
##     80        0.7183             nan     0.1000    0.0002
##    100        0.6850             nan     0.1000   -0.0007
##    120        0.6578             nan     0.1000   -0.0006
##    140        0.6382             nan     0.1000   -0.0011
##    150        0.6291             nan     0.1000   -0.0012
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2493             nan     0.1000    0.0236
##      2        1.2040             nan     0.1000    0.0211
##      3        1.1584             nan     0.1000    0.0179
##      4        1.1219             nan     0.1000    0.0189
##      5        1.0897             nan     0.1000    0.0137
##      6        1.0677             nan     0.1000    0.0087
##      7        1.0367             nan     0.1000    0.0148
##      8        1.0092             nan     0.1000    0.0122
##      9        0.9895             nan     0.1000    0.0078
##     10        0.9693             nan     0.1000    0.0091
##     20        0.8287             nan     0.1000    0.0013
##     40        0.7065             nan     0.1000   -0.0003
##     60        0.6430             nan     0.1000   -0.0012
##     80        0.5885             nan     0.1000   -0.0009
##    100        0.5544             nan     0.1000   -0.0024
##    120        0.5190             nan     0.1000   -0.0013
##    140        0.4901             nan     0.1000   -0.0010
##    150        0.4738             nan     0.1000   -0.0004
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2264             nan     0.1000    0.0315
##      2        1.1721             nan     0.1000    0.0231
##      3        1.1255             nan     0.1000    0.0183
##      4        1.0867             nan     0.1000    0.0131
##      5        1.0473             nan     0.1000    0.0185
##      6        1.0139             nan     0.1000    0.0131
##      7        0.9784             nan     0.1000    0.0147
##      8        0.9516             nan     0.1000    0.0101
##      9        0.9333             nan     0.1000    0.0066
##     10        0.9077             nan     0.1000    0.0094
##     20        0.7627             nan     0.1000    0.0022
##     40        0.6261             nan     0.1000    0.0009
##     60        0.5488             nan     0.1000   -0.0019
##     80        0.4834             nan     0.1000   -0.0010
##    100        0.4273             nan     0.1000   -0.0005
##    120        0.3922             nan     0.1000    0.0001
##    140        0.3604             nan     0.1000   -0.0010
##    150        0.3436             nan     0.1000   -0.0015
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2634             nan     0.1000    0.0125
##      2        1.2366             nan     0.1000    0.0083
##      3        1.2139             nan     0.1000    0.0112
##      4        1.1952             nan     0.1000    0.0072
##      5        1.1780             nan     0.1000    0.0069
##      6        1.1646             nan     0.1000    0.0040
##      7        1.1464             nan     0.1000    0.0055
##      8        1.1342             nan     0.1000    0.0041
##      9        1.1184             nan     0.1000    0.0065
##     10        1.1085             nan     0.1000    0.0031
##     20        1.0155             nan     0.1000    0.0027
##     40        0.9164             nan     0.1000   -0.0012
##     60        0.8576             nan     0.1000    0.0005
##     80        0.8189             nan     0.1000    0.0001
##    100        0.7879             nan     0.1000   -0.0003
##    120        0.7578             nan     0.1000   -0.0005
##    140        0.7388             nan     0.1000   -0.0009
##    150        0.7306             nan     0.1000   -0.0011
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2520             nan     0.1000    0.0195
##      2        1.2139             nan     0.1000    0.0167
##      3        1.1844             nan     0.1000    0.0127
##      4        1.1571             nan     0.1000    0.0123
##      5        1.1327             nan     0.1000    0.0100
##      6        1.1074             nan     0.1000    0.0091
##      7        1.0872             nan     0.1000    0.0083
##      8        1.0704             nan     0.1000    0.0070
##      9        1.0531             nan     0.1000    0.0053
##     10        1.0344             nan     0.1000    0.0072
##     20        0.9160             nan     0.1000    0.0044
##     40        0.7894             nan     0.1000    0.0007
##     60        0.7221             nan     0.1000   -0.0012
##     80        0.6653             nan     0.1000   -0.0032
##    100        0.6300             nan     0.1000   -0.0009
##    120        0.5908             nan     0.1000   -0.0024
##    140        0.5511             nan     0.1000   -0.0009
##    150        0.5320             nan     0.1000   -0.0021
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2422             nan     0.1000    0.0224
##      2        1.1999             nan     0.1000    0.0166
##      3        1.1602             nan     0.1000    0.0155
##      4        1.1258             nan     0.1000    0.0099
##      5        1.0998             nan     0.1000    0.0085
##      6        1.0743             nan     0.1000    0.0117
##      7        1.0476             nan     0.1000    0.0133
##      8        1.0242             nan     0.1000    0.0071
##      9        1.0065             nan     0.1000    0.0050
##     10        0.9843             nan     0.1000    0.0068
##     20        0.8498             nan     0.1000   -0.0008
##     40        0.7109             nan     0.1000   -0.0018
##     60        0.6267             nan     0.1000    0.0002
##     80        0.5567             nan     0.1000   -0.0004
##    100        0.5001             nan     0.1000   -0.0012
##    120        0.4479             nan     0.1000   -0.0007
##    140        0.4086             nan     0.1000   -0.0003
##    150        0.3880             nan     0.1000   -0.0010
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2782             nan     0.1000    0.0127
##      2        1.2534             nan     0.1000    0.0120
##      3        1.2285             nan     0.1000    0.0064
##      4        1.2020             nan     0.1000    0.0105
##      5        1.1796             nan     0.1000    0.0080
##      6        1.1620             nan     0.1000    0.0091
##      7        1.1449             nan     0.1000    0.0065
##      8        1.1254             nan     0.1000    0.0084
##      9        1.1103             nan     0.1000    0.0070
##     10        1.0955             nan     0.1000    0.0049
##     20        0.9903             nan     0.1000    0.0019
##     40        0.8997             nan     0.1000    0.0000
##     60        0.8557             nan     0.1000   -0.0017
##     80        0.8272             nan     0.1000   -0.0013
##    100        0.8016             nan     0.1000   -0.0003
##    120        0.7825             nan     0.1000   -0.0001
##    140        0.7672             nan     0.1000   -0.0008
##    150        0.7586             nan     0.1000   -0.0002
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2642             nan     0.1000    0.0123
##      2        1.2217             nan     0.1000    0.0203
##      3        1.1853             nan     0.1000    0.0161
##      4        1.1510             nan     0.1000    0.0141
##      5        1.1234             nan     0.1000    0.0126
##      6        1.1000             nan     0.1000    0.0117
##      7        1.0750             nan     0.1000    0.0112
##      8        1.0574             nan     0.1000    0.0082
##      9        1.0398             nan     0.1000    0.0086
##     10        1.0224             nan     0.1000    0.0072
##     20        0.9115             nan     0.1000    0.0025
##     40        0.8141             nan     0.1000   -0.0014
##     60        0.7526             nan     0.1000   -0.0008
##     80        0.7045             nan     0.1000   -0.0037
##    100        0.6551             nan     0.1000   -0.0012
##    120        0.6241             nan     0.1000   -0.0009
##    140        0.5920             nan     0.1000    0.0003
##    150        0.5752             nan     0.1000   -0.0030
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2593             nan     0.1000    0.0203
##      2        1.2084             nan     0.1000    0.0212
##      3        1.1652             nan     0.1000    0.0178
##      4        1.1229             nan     0.1000    0.0175
##      5        1.0892             nan     0.1000    0.0150
##      6        1.0622             nan     0.1000    0.0117
##      7        1.0352             nan     0.1000    0.0114
##      8        1.0110             nan     0.1000    0.0087
##      9        0.9928             nan     0.1000    0.0064
##     10        0.9764             nan     0.1000    0.0045
##     20        0.8430             nan     0.1000    0.0005
##     40        0.7280             nan     0.1000   -0.0020
##     60        0.6378             nan     0.1000    0.0002
##     80        0.5829             nan     0.1000   -0.0006
##    100        0.5282             nan     0.1000   -0.0007
##    120        0.4821             nan     0.1000   -0.0010
##    140        0.4372             nan     0.1000   -0.0011
##    150        0.4154             nan     0.1000   -0.0023
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2789             nan     0.1000    0.0200
##      2        1.2388             nan     0.1000    0.0157
##      3        1.2110             nan     0.1000    0.0142
##      4        1.1846             nan     0.1000    0.0098
##      5        1.1648             nan     0.1000    0.0085
##      6        1.1448             nan     0.1000    0.0090
##      7        1.1263             nan     0.1000    0.0065
##      8        1.1111             nan     0.1000    0.0081
##      9        1.0975             nan     0.1000    0.0066
##     10        1.0801             nan     0.1000    0.0061
##     20        0.9730             nan     0.1000    0.0030
##     40        0.8593             nan     0.1000    0.0010
##     60        0.7969             nan     0.1000   -0.0009
##     80        0.7634             nan     0.1000   -0.0004
##    100        0.7339             nan     0.1000   -0.0009
##    120        0.7092             nan     0.1000   -0.0010
##    140        0.6900             nan     0.1000   -0.0003
##    150        0.6821             nan     0.1000   -0.0002
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2691             nan     0.1000    0.0250
##      2        1.2231             nan     0.1000    0.0186
##      3        1.1816             nan     0.1000    0.0179
##      4        1.1482             nan     0.1000    0.0137
##      5        1.1181             nan     0.1000    0.0137
##      6        1.0913             nan     0.1000    0.0106
##      7        1.0677             nan     0.1000    0.0086
##      8        1.0489             nan     0.1000    0.0067
##      9        1.0314             nan     0.1000    0.0047
##     10        1.0146             nan     0.1000    0.0045
##     20        0.8684             nan     0.1000    0.0021
##     40        0.7535             nan     0.1000    0.0012
##     60        0.6814             nan     0.1000   -0.0010
##     80        0.6201             nan     0.1000    0.0007
##    100        0.5640             nan     0.1000   -0.0011
##    120        0.5307             nan     0.1000    0.0002
##    140        0.4928             nan     0.1000    0.0005
##    150        0.4776             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2590             nan     0.1000    0.0245
##      2        1.1928             nan     0.1000    0.0280
##      3        1.1410             nan     0.1000    0.0188
##      4        1.1064             nan     0.1000    0.0155
##      5        1.0703             nan     0.1000    0.0153
##      6        1.0461             nan     0.1000    0.0088
##      7        1.0169             nan     0.1000    0.0106
##      8        0.9875             nan     0.1000    0.0139
##      9        0.9624             nan     0.1000    0.0109
##     10        0.9430             nan     0.1000    0.0040
##     20        0.8068             nan     0.1000    0.0046
##     40        0.6647             nan     0.1000    0.0000
##     60        0.5730             nan     0.1000   -0.0009
##     80        0.5041             nan     0.1000   -0.0009
##    100        0.4555             nan     0.1000   -0.0020
##    120        0.4125             nan     0.1000   -0.0005
##    140        0.3729             nan     0.1000   -0.0005
##    150        0.3554             nan     0.1000   -0.0002
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2436             nan     0.1000    0.0201
##      2        1.2065             nan     0.1000    0.0155
##      3        1.1736             nan     0.1000    0.0135
##      4        1.1476             nan     0.1000    0.0130
##      5        1.1213             nan     0.1000    0.0110
##      6        1.0980             nan     0.1000    0.0096
##      7        1.0783             nan     0.1000    0.0077
##      8        1.0592             nan     0.1000    0.0066
##      9        1.0437             nan     0.1000    0.0065
##     10        1.0279             nan     0.1000    0.0059
##     20        0.9196             nan     0.1000    0.0041
##     40        0.8246             nan     0.1000    0.0008
##     60        0.7798             nan     0.1000   -0.0009
##     80        0.7499             nan     0.1000   -0.0017
##    100        0.7305             nan     0.1000   -0.0006
##    120        0.7143             nan     0.1000   -0.0007
##    140        0.6980             nan     0.1000   -0.0006
##    150        0.6921             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2222             nan     0.1000    0.0253
##      2        1.1776             nan     0.1000    0.0205
##      3        1.1347             nan     0.1000    0.0212
##      4        1.1013             nan     0.1000    0.0159
##      5        1.0692             nan     0.1000    0.0137
##      6        1.0337             nan     0.1000    0.0129
##      7        1.0070             nan     0.1000    0.0119
##      8        0.9844             nan     0.1000    0.0087
##      9        0.9614             nan     0.1000    0.0100
##     10        0.9420             nan     0.1000    0.0070
##     20        0.8237             nan     0.1000    0.0011
##     40        0.7267             nan     0.1000    0.0012
##     60        0.6674             nan     0.1000   -0.0006
##     80        0.6127             nan     0.1000   -0.0005
##    100        0.5756             nan     0.1000   -0.0032
##    120        0.5396             nan     0.1000   -0.0007
##    140        0.5045             nan     0.1000    0.0010
##    150        0.4905             nan     0.1000   -0.0005
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2132             nan     0.1000    0.0320
##      2        1.1602             nan     0.1000    0.0252
##      3        1.1178             nan     0.1000    0.0208
##      4        1.0743             nan     0.1000    0.0192
##      5        1.0353             nan     0.1000    0.0182
##      6        1.0024             nan     0.1000    0.0142
##      7        0.9826             nan     0.1000    0.0066
##      8        0.9542             nan     0.1000    0.0113
##      9        0.9291             nan     0.1000    0.0114
##     10        0.9083             nan     0.1000    0.0090
##     20        0.7794             nan     0.1000   -0.0013
##     40        0.6690             nan     0.1000   -0.0013
##     60        0.5880             nan     0.1000    0.0005
##     80        0.5262             nan     0.1000   -0.0021
##    100        0.4818             nan     0.1000   -0.0027
##    120        0.4359             nan     0.1000   -0.0005
##    140        0.3978             nan     0.1000   -0.0006
##    150        0.3757             nan     0.1000   -0.0004
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2568             nan     0.1000    0.0165
##      2        1.2267             nan     0.1000    0.0116
##      3        1.2035             nan     0.1000    0.0103
##      4        1.1801             nan     0.1000    0.0099
##      5        1.1550             nan     0.1000    0.0117
##      6        1.1347             nan     0.1000    0.0094
##      7        1.1123             nan     0.1000    0.0101
##      8        1.0944             nan     0.1000    0.0071
##      9        1.0768             nan     0.1000    0.0060
##     10        1.0599             nan     0.1000    0.0079
##     20        0.9458             nan     0.1000    0.0024
##     40        0.8225             nan     0.1000    0.0004
##     60        0.7595             nan     0.1000    0.0007
##     80        0.7241             nan     0.1000   -0.0001
##    100        0.6884             nan     0.1000   -0.0006
##    120        0.6598             nan     0.1000    0.0000
##    140        0.6432             nan     0.1000   -0.0009
##    150        0.6311             nan     0.1000   -0.0007
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2417             nan     0.1000    0.0215
##      2        1.1974             nan     0.1000    0.0201
##      3        1.1613             nan     0.1000    0.0180
##      4        1.1194             nan     0.1000    0.0155
##      5        1.0929             nan     0.1000    0.0086
##      6        1.0655             nan     0.1000    0.0107
##      7        1.0418             nan     0.1000    0.0085
##      8        1.0172             nan     0.1000    0.0104
##      9        0.9931             nan     0.1000    0.0100
##     10        0.9769             nan     0.1000    0.0026
##     20        0.8336             nan     0.1000    0.0033
##     40        0.7080             nan     0.1000    0.0012
##     60        0.6319             nan     0.1000   -0.0012
##     80        0.5870             nan     0.1000   -0.0004
##    100        0.5433             nan     0.1000   -0.0009
##    120        0.5094             nan     0.1000   -0.0006
##    140        0.4816             nan     0.1000   -0.0017
##    150        0.4636             nan     0.1000   -0.0014
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2329             nan     0.1000    0.0279
##      2        1.1825             nan     0.1000    0.0229
##      3        1.1362             nan     0.1000    0.0214
##      4        1.0914             nan     0.1000    0.0184
##      5        1.0456             nan     0.1000    0.0198
##      6        1.0104             nan     0.1000    0.0165
##      7        0.9780             nan     0.1000    0.0125
##      8        0.9578             nan     0.1000    0.0066
##      9        0.9310             nan     0.1000    0.0114
##     10        0.9107             nan     0.1000    0.0093
##     20        0.7713             nan     0.1000    0.0035
##     40        0.6200             nan     0.1000    0.0010
##     60        0.5375             nan     0.1000   -0.0005
##     80        0.4820             nan     0.1000   -0.0010
##    100        0.4328             nan     0.1000   -0.0004
##    120        0.3833             nan     0.1000   -0.0008
##    140        0.3506             nan     0.1000   -0.0009
##    150        0.3385             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2609             nan     0.1000    0.0169
##      2        1.2312             nan     0.1000    0.0129
##      3        1.2029             nan     0.1000    0.0120
##      4        1.1815             nan     0.1000    0.0097
##      5        1.1573             nan     0.1000    0.0080
##      6        1.1349             nan     0.1000    0.0108
##      7        1.1190             nan     0.1000    0.0055
##      8        1.1104             nan     0.1000    0.0023
##      9        1.0985             nan     0.1000    0.0035
##     10        1.0868             nan     0.1000    0.0034
##     20        0.9885             nan     0.1000    0.0037
##     40        0.8822             nan     0.1000   -0.0001
##     60        0.8275             nan     0.1000   -0.0006
##     80        0.7855             nan     0.1000    0.0011
##    100        0.7504             nan     0.1000   -0.0003
##    120        0.7280             nan     0.1000   -0.0002
##    140        0.7046             nan     0.1000   -0.0002
##    150        0.6945             nan     0.1000    0.0006
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2519             nan     0.1000    0.0199
##      2        1.2139             nan     0.1000    0.0180
##      3        1.1725             nan     0.1000    0.0176
##      4        1.1418             nan     0.1000    0.0120
##      5        1.1130             nan     0.1000    0.0116
##      6        1.0870             nan     0.1000    0.0129
##      7        1.0667             nan     0.1000    0.0094
##      8        1.0470             nan     0.1000    0.0062
##      9        1.0283             nan     0.1000    0.0067
##     10        1.0153             nan     0.1000    0.0026
##     20        0.9005             nan     0.1000    0.0016
##     40        0.7712             nan     0.1000    0.0023
##     60        0.6961             nan     0.1000   -0.0006
##     80        0.6427             nan     0.1000    0.0002
##    100        0.5955             nan     0.1000   -0.0005
##    120        0.5556             nan     0.1000   -0.0002
##    140        0.5267             nan     0.1000   -0.0013
##    150        0.5102             nan     0.1000    0.0000
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2389             nan     0.1000    0.0257
##      2        1.2016             nan     0.1000    0.0162
##      3        1.1631             nan     0.1000    0.0162
##      4        1.1238             nan     0.1000    0.0177
##      5        1.0851             nan     0.1000    0.0141
##      6        1.0518             nan     0.1000    0.0152
##      7        1.0273             nan     0.1000    0.0080
##      8        1.0077             nan     0.1000    0.0070
##      9        0.9879             nan     0.1000    0.0075
##     10        0.9667             nan     0.1000    0.0094
##     20        0.8262             nan     0.1000    0.0037
##     40        0.6861             nan     0.1000    0.0012
##     60        0.6028             nan     0.1000   -0.0012
##     80        0.5415             nan     0.1000    0.0001
##    100        0.4907             nan     0.1000   -0.0012
##    120        0.4470             nan     0.1000   -0.0006
##    140        0.4088             nan     0.1000   -0.0013
##    150        0.3939             nan     0.1000   -0.0020
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2563             nan     0.1000    0.0166
##      2        1.2269             nan     0.1000    0.0128
##      3        1.2003             nan     0.1000    0.0115
##      4        1.1788             nan     0.1000    0.0089
##      5        1.1604             nan     0.1000    0.0074
##      6        1.1380             nan     0.1000    0.0102
##      7        1.1231             nan     0.1000    0.0069
##      8        1.1038             nan     0.1000    0.0081
##      9        1.0887             nan     0.1000    0.0057
##     10        1.0736             nan     0.1000    0.0059
##     20        0.9776             nan     0.1000    0.0009
##     40        0.8959             nan     0.1000    0.0004
##     60        0.8459             nan     0.1000    0.0006
##     80        0.8084             nan     0.1000   -0.0016
##    100        0.7821             nan     0.1000   -0.0004
##    120        0.7659             nan     0.1000   -0.0016
##    140        0.7463             nan     0.1000   -0.0030
##    150        0.7349             nan     0.1000   -0.0002
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2383             nan     0.1000    0.0225
##      2        1.2018             nan     0.1000    0.0137
##      3        1.1596             nan     0.1000    0.0193
##      4        1.1261             nan     0.1000    0.0133
##      5        1.1002             nan     0.1000    0.0118
##      6        1.0785             nan     0.1000    0.0080
##      7        1.0506             nan     0.1000    0.0098
##      8        1.0344             nan     0.1000    0.0045
##      9        1.0171             nan     0.1000    0.0067
##     10        1.0020             nan     0.1000    0.0056
##     20        0.8889             nan     0.1000    0.0008
##     40        0.7821             nan     0.1000   -0.0007
##     60        0.7061             nan     0.1000   -0.0008
##     80        0.6534             nan     0.1000   -0.0017
##    100        0.6159             nan     0.1000   -0.0002
##    120        0.5770             nan     0.1000   -0.0001
##    140        0.5446             nan     0.1000   -0.0004
##    150        0.5337             nan     0.1000   -0.0004
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2360             nan     0.1000    0.0206
##      2        1.1917             nan     0.1000    0.0169
##      3        1.1432             nan     0.1000    0.0183
##      4        1.1081             nan     0.1000    0.0091
##      5        1.0792             nan     0.1000    0.0119
##      6        1.0566             nan     0.1000    0.0076
##      7        1.0364             nan     0.1000    0.0055
##      8        1.0114             nan     0.1000    0.0086
##      9        0.9937             nan     0.1000    0.0045
##     10        0.9720             nan     0.1000    0.0093
##     20        0.8454             nan     0.1000    0.0010
##     40        0.7206             nan     0.1000    0.0015
##     60        0.6329             nan     0.1000   -0.0021
##     80        0.5678             nan     0.1000    0.0006
##    100        0.5141             nan     0.1000   -0.0020
##    120        0.4736             nan     0.1000   -0.0011
##    140        0.4390             nan     0.1000   -0.0008
##    150        0.4218             nan     0.1000   -0.0014
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2648             nan     0.1000    0.0180
##      2        1.2312             nan     0.1000    0.0137
##      3        1.2068             nan     0.1000    0.0103
##      4        1.1821             nan     0.1000    0.0117
##      5        1.1600             nan     0.1000    0.0099
##      6        1.1389             nan     0.1000    0.0083
##      7        1.1196             nan     0.1000    0.0068
##      8        1.1002             nan     0.1000    0.0067
##      9        1.0819             nan     0.1000    0.0084
##     10        1.0663             nan     0.1000    0.0048
##     20        0.9508             nan     0.1000    0.0035
##     40        0.8332             nan     0.1000    0.0008
##     60        0.7786             nan     0.1000   -0.0007
##     80        0.7404             nan     0.1000    0.0003
##    100        0.7162             nan     0.1000   -0.0001
##    120        0.6991             nan     0.1000   -0.0005
##    140        0.6807             nan     0.1000   -0.0004
##    150        0.6713             nan     0.1000   -0.0013
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2467             nan     0.1000    0.0222
##      2        1.1930             nan     0.1000    0.0216
##      3        1.1597             nan     0.1000    0.0156
##      4        1.1297             nan     0.1000    0.0123
##      5        1.1045             nan     0.1000    0.0080
##      6        1.0734             nan     0.1000    0.0130
##      7        1.0457             nan     0.1000    0.0131
##      8        1.0195             nan     0.1000    0.0099
##      9        0.9926             nan     0.1000    0.0103
##     10        0.9715             nan     0.1000    0.0091
##     20        0.8481             nan     0.1000    0.0025
##     40        0.7336             nan     0.1000   -0.0009
##     60        0.6640             nan     0.1000    0.0003
##     80        0.6135             nan     0.1000   -0.0008
##    100        0.5759             nan     0.1000   -0.0011
##    120        0.5450             nan     0.1000   -0.0010
##    140        0.5113             nan     0.1000   -0.0016
##    150        0.4968             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2324             nan     0.1000    0.0277
##      2        1.1793             nan     0.1000    0.0256
##      3        1.1341             nan     0.1000    0.0182
##      4        1.0928             nan     0.1000    0.0187
##      5        1.0558             nan     0.1000    0.0150
##      6        1.0260             nan     0.1000    0.0114
##      7        0.9984             nan     0.1000    0.0086
##      8        0.9725             nan     0.1000    0.0115
##      9        0.9481             nan     0.1000    0.0074
##     10        0.9310             nan     0.1000    0.0066
##     20        0.7871             nan     0.1000    0.0029
##     40        0.6495             nan     0.1000   -0.0012
##     60        0.5733             nan     0.1000   -0.0012
##     80        0.5160             nan     0.1000   -0.0010
##    100        0.4648             nan     0.1000    0.0004
##    120        0.4265             nan     0.1000   -0.0008
##    140        0.3894             nan     0.1000   -0.0007
##    150        0.3758             nan     0.1000   -0.0016
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2025             nan     0.1000    0.0215
##      2        1.1719             nan     0.1000    0.0154
##      3        1.1426             nan     0.1000    0.0131
##      4        1.1151             nan     0.1000    0.0099
##      5        1.0900             nan     0.1000    0.0091
##      6        1.0635             nan     0.1000    0.0124
##      7        1.0400             nan     0.1000    0.0091
##      8        1.0235             nan     0.1000    0.0062
##      9        1.0086             nan     0.1000    0.0052
##     10        0.9924             nan     0.1000    0.0073
##     20        0.8834             nan     0.1000    0.0041
##     40        0.7690             nan     0.1000   -0.0001
##     60        0.7087             nan     0.1000   -0.0004
##     80        0.6679             nan     0.1000   -0.0024
##    100        0.6371             nan     0.1000   -0.0001
##    120        0.6145             nan     0.1000   -0.0007
##    140        0.5955             nan     0.1000   -0.0005
##    150        0.5871             nan     0.1000   -0.0006
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.1767             nan     0.1000    0.0301
##      2        1.1290             nan     0.1000    0.0231
##      3        1.0853             nan     0.1000    0.0181
##      4        1.0528             nan     0.1000    0.0131
##      5        1.0167             nan     0.1000    0.0163
##      6        0.9869             nan     0.1000    0.0147
##      7        0.9596             nan     0.1000    0.0115
##      8        0.9401             nan     0.1000    0.0098
##      9        0.9229             nan     0.1000    0.0053
##     10        0.8995             nan     0.1000    0.0088
##     20        0.7653             nan     0.1000    0.0023
##     40        0.6421             nan     0.1000   -0.0013
##     60        0.5662             nan     0.1000    0.0011
##     80        0.5132             nan     0.1000   -0.0010
##    100        0.4712             nan     0.1000    0.0001
##    120        0.4390             nan     0.1000   -0.0008
##    140        0.4068             nan     0.1000    0.0000
##    150        0.3905             nan     0.1000   -0.0013
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.1769             nan     0.1000    0.0318
##      2        1.1159             nan     0.1000    0.0280
##      3        1.0755             nan     0.1000    0.0168
##      4        1.0260             nan     0.1000    0.0187
##      5        0.9906             nan     0.1000    0.0145
##      6        0.9568             nan     0.1000    0.0117
##      7        0.9253             nan     0.1000    0.0126
##      8        0.9017             nan     0.1000    0.0079
##      9        0.8775             nan     0.1000    0.0077
##     10        0.8617             nan     0.1000    0.0049
##     20        0.7064             nan     0.1000    0.0005
##     40        0.5585             nan     0.1000   -0.0010
##     60        0.4777             nan     0.1000   -0.0019
##     80        0.4134             nan     0.1000   -0.0003
##    100        0.3644             nan     0.1000   -0.0003
##    120        0.3244             nan     0.1000   -0.0018
##    140        0.2873             nan     0.1000   -0.0007
##    150        0.2734             nan     0.1000   -0.0009
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2284             nan     0.1000    0.0183
##      2        1.1931             nan     0.1000    0.0148
##      3        1.1676             nan     0.1000    0.0129
##      4        1.1426             nan     0.1000    0.0107
##      5        1.1228             nan     0.1000    0.0084
##      6        1.1029             nan     0.1000    0.0082
##      7        1.0870             nan     0.1000    0.0075
##      8        1.0685             nan     0.1000    0.0080
##      9        1.0492             nan     0.1000    0.0073
##     10        1.0378             nan     0.1000    0.0056
##     20        0.9290             nan     0.1000    0.0022
##     40        0.8378             nan     0.1000   -0.0012
##     60        0.7934             nan     0.1000   -0.0005
##     80        0.7569             nan     0.1000   -0.0001
##    100        0.7296             nan     0.1000   -0.0003
##    120        0.7053             nan     0.1000   -0.0010
##    140        0.6888             nan     0.1000   -0.0020
##    150        0.6816             nan     0.1000    0.0000
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2134             nan     0.1000    0.0266
##      2        1.1647             nan     0.1000    0.0193
##      3        1.1292             nan     0.1000    0.0166
##      4        1.1002             nan     0.1000    0.0144
##      5        1.0717             nan     0.1000    0.0132
##      6        1.0460             nan     0.1000    0.0111
##      7        1.0206             nan     0.1000    0.0086
##      8        0.9947             nan     0.1000    0.0078
##      9        0.9768             nan     0.1000    0.0073
##     10        0.9594             nan     0.1000    0.0082
##     20        0.8513             nan     0.1000    0.0007
##     40        0.7508             nan     0.1000   -0.0008
##     60        0.6899             nan     0.1000   -0.0009
##     80        0.6505             nan     0.1000   -0.0028
##    100        0.6084             nan     0.1000   -0.0009
##    120        0.5730             nan     0.1000    0.0003
##    140        0.5397             nan     0.1000   -0.0012
##    150        0.5235             nan     0.1000   -0.0010
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2042             nan     0.1000    0.0309
##      2        1.1517             nan     0.1000    0.0223
##      3        1.1126             nan     0.1000    0.0185
##      4        1.0742             nan     0.1000    0.0110
##      5        1.0387             nan     0.1000    0.0169
##      6        1.0125             nan     0.1000    0.0100
##      7        0.9804             nan     0.1000    0.0128
##      8        0.9528             nan     0.1000    0.0104
##      9        0.9367             nan     0.1000    0.0030
##     10        0.9196             nan     0.1000    0.0067
##     20        0.8011             nan     0.1000    0.0005
##     40        0.6806             nan     0.1000   -0.0030
##     60        0.6035             nan     0.1000   -0.0007
##     80        0.5338             nan     0.1000    0.0001
##    100        0.4840             nan     0.1000    0.0001
##    120        0.4423             nan     0.1000   -0.0001
##    140        0.4054             nan     0.1000   -0.0024
##    150        0.3857             nan     0.1000   -0.0003
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2318             nan     0.1000    0.0202
##      2        1.2005             nan     0.1000    0.0169
##      3        1.1628             nan     0.1000    0.0157
##      4        1.1381             nan     0.1000    0.0108
##      5        1.1145             nan     0.1000    0.0115
##      6        1.0978             nan     0.1000    0.0066
##      7        1.0756             nan     0.1000    0.0087
##      8        1.0600             nan     0.1000    0.0047
##      9        1.0400             nan     0.1000    0.0078
##     10        1.0250             nan     0.1000    0.0059
##     20        0.9103             nan     0.1000    0.0031
##     40        0.8089             nan     0.1000    0.0005
##     60        0.7597             nan     0.1000    0.0003
##     80        0.7312             nan     0.1000   -0.0015
##    100        0.7104             nan     0.1000   -0.0010
##    120        0.6934             nan     0.1000   -0.0006
##    140        0.6802             nan     0.1000   -0.0013
##    150        0.6738             nan     0.1000   -0.0007
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2229             nan     0.1000    0.0229
##      2        1.1765             nan     0.1000    0.0215
##      3        1.1334             nan     0.1000    0.0206
##      4        1.0974             nan     0.1000    0.0129
##      5        1.0633             nan     0.1000    0.0143
##      6        1.0397             nan     0.1000    0.0093
##      7        1.0134             nan     0.1000    0.0083
##      8        0.9956             nan     0.1000    0.0057
##      9        0.9749             nan     0.1000    0.0054
##     10        0.9524             nan     0.1000    0.0069
##     20        0.8266             nan     0.1000    0.0019
##     40        0.7255             nan     0.1000   -0.0004
##     60        0.6666             nan     0.1000    0.0003
##     80        0.6120             nan     0.1000   -0.0001
##    100        0.5734             nan     0.1000    0.0000
##    120        0.5368             nan     0.1000   -0.0008
##    140        0.5100             nan     0.1000   -0.0012
##    150        0.4936             nan     0.1000   -0.0014
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2121             nan     0.1000    0.0271
##      2        1.1583             nan     0.1000    0.0214
##      3        1.1084             nan     0.1000    0.0232
##      4        1.0725             nan     0.1000    0.0151
##      5        1.0381             nan     0.1000    0.0134
##      6        1.0046             nan     0.1000    0.0161
##      7        0.9742             nan     0.1000    0.0136
##      8        0.9481             nan     0.1000    0.0090
##      9        0.9263             nan     0.1000    0.0076
##     10        0.9084             nan     0.1000    0.0043
##     20        0.7616             nan     0.1000    0.0015
##     40        0.6442             nan     0.1000   -0.0013
##     60        0.5641             nan     0.1000   -0.0003
##     80        0.4952             nan     0.1000   -0.0010
##    100        0.4487             nan     0.1000   -0.0012
##    120        0.4094             nan     0.1000   -0.0019
##    140        0.3703             nan     0.1000   -0.0010
##    150        0.3519             nan     0.1000   -0.0001
## 
## Iter   TrainDeviance   ValidDeviance   StepSize   Improve
##      1        1.2539             nan     0.1000    0.0143
##      2        1.2203             nan     0.1000    0.0123
##      3        1.1915             nan     0.1000    0.0086
##      4        1.1714             nan     0.1000    0.0071
##      5        1.1518             nan     0.1000    0.0082
##      6        1.1367             nan     0.1000    0.0050
##      7        1.1164             nan     0.1000    0.0078
##      8        1.0996             nan     0.1000    0.0070
##      9        1.0862             nan     0.1000    0.0049
##     10        1.0735             nan     0.1000    0.0053
##     20        0.9764             nan     0.1000    0.0005
##     40        0.8873             nan     0.1000    0.0007
##     50        0.8661             nan     0.1000   -0.0020
model_gbm
## Stochastic Gradient Boosting 
## 
## 538 samples
##   8 predictor
##   2 classes: 'neg', 'pos' 
## 
## No pre-processing
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... 
## Resampling results across tuning parameters:
## 
##   interaction.depth  n.trees  Accuracy   Kappa    
##   1                   50      0.7581561  0.4428866
##   1                  100      0.7575227  0.4488587
##   1                  150      0.7531729  0.4404393
##   2                   50      0.7561162  0.4433477
##   2                  100      0.7484656  0.4298475
##   2                  150      0.7432515  0.4181766
##   3                   50      0.7562096  0.4490012
##   3                  100      0.7474033  0.4302205
##   3                  150      0.7441665  0.4238704
## 
## Tuning parameter 'shrinkage' was held constant at a value of 0.1
## 
## Tuning parameter 'n.minobsinnode' was held constant at a value of 10
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were n.trees = 50, interaction.depth
##  = 1, shrinkage = 0.1 and n.minobsinnode = 10.

Gradient Boosting Machine

model_xgb <- train(diabetes ~ ., data = diabetes_train1, method = "xgbTree")
model_xgb
## eXtreme Gradient Boosting 
## 
## 538 samples
##   8 predictor
##   2 classes: 'neg', 'pos' 
## 
## No pre-processing
## Resampling: Bootstrapped (25 reps) 
## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... 
## Resampling results across tuning parameters:
## 
##   eta  max_depth  colsample_bytree  subsample  nrounds  Accuracy 
##   0.3  1          0.6               0.50        50      0.7543059
##   0.3  1          0.6               0.50       100      0.7514498
##   0.3  1          0.6               0.50       150      0.7401059
##   0.3  1          0.6               0.75        50      0.7629041
##   0.3  1          0.6               0.75       100      0.7596995
##   0.3  1          0.6               0.75       150      0.7528011
##   0.3  1          0.6               1.00        50      0.7609537
##   0.3  1          0.6               1.00       100      0.7582121
##   0.3  1          0.6               1.00       150      0.7499080
##   0.3  1          0.8               0.50        50      0.7595069
##   0.3  1          0.8               0.50       100      0.7468614
##   0.3  1          0.8               0.50       150      0.7458352
##   0.3  1          0.8               0.75        50      0.7612828
##   0.3  1          0.8               0.75       100      0.7504202
##   0.3  1          0.8               0.75       150      0.7451137
##   0.3  1          0.8               1.00        50      0.7615568
##   0.3  1          0.8               1.00       100      0.7575015
##   0.3  1          0.8               1.00       150      0.7529885
##   0.3  2          0.6               0.50        50      0.7445060
##   0.3  2          0.6               0.50       100      0.7324920
##   0.3  2          0.6               0.50       150      0.7248227
##   0.3  2          0.6               0.75        50      0.7502667
##   0.3  2          0.6               0.75       100      0.7414999
##   0.3  2          0.6               0.75       150      0.7308230
##   0.3  2          0.6               1.00        50      0.7497999
##   0.3  2          0.6               1.00       100      0.7341151
##   0.3  2          0.6               1.00       150      0.7307433
##   0.3  2          0.8               0.50        50      0.7463895
##   0.3  2          0.8               0.50       100      0.7310613
##   0.3  2          0.8               0.50       150      0.7255107
##   0.3  2          0.8               0.75        50      0.7445795
##   0.3  2          0.8               0.75       100      0.7349672
##   0.3  2          0.8               0.75       150      0.7294988
##   0.3  2          0.8               1.00        50      0.7505633
##   0.3  2          0.8               1.00       100      0.7409459
##   0.3  2          0.8               1.00       150      0.7330636
##   0.3  3          0.6               0.50        50      0.7449172
##   0.3  3          0.6               0.50       100      0.7327773
##   0.3  3          0.6               0.50       150      0.7290412
##   0.3  3          0.6               0.75        50      0.7341227
##   0.3  3          0.6               0.75       100      0.7208463
##   0.3  3          0.6               0.75       150      0.7195923
##   0.3  3          0.6               1.00        50      0.7390111
##   0.3  3          0.6               1.00       100      0.7326315
##   0.3  3          0.6               1.00       150      0.7283834
##   0.3  3          0.8               0.50        50      0.7339048
##   0.3  3          0.8               0.50       100      0.7253577
##   0.3  3          0.8               0.50       150      0.7249229
##   0.3  3          0.8               0.75        50      0.7382761
##   0.3  3          0.8               0.75       100      0.7275341
##   0.3  3          0.8               0.75       150      0.7253066
##   0.3  3          0.8               1.00        50      0.7441233
##   0.3  3          0.8               1.00       100      0.7384999
##   0.3  3          0.8               1.00       150      0.7325560
##   0.4  1          0.6               0.50        50      0.7594764
##   0.4  1          0.6               0.50       100      0.7499864
##   0.4  1          0.6               0.50       150      0.7393398
##   0.4  1          0.6               0.75        50      0.7583405
##   0.4  1          0.6               0.75       100      0.7484413
##   0.4  1          0.6               0.75       150      0.7431933
##   0.4  1          0.6               1.00        50      0.7638681
##   0.4  1          0.6               1.00       100      0.7542095
##   0.4  1          0.6               1.00       150      0.7467963
##   0.4  1          0.8               0.50        50      0.7560829
##   0.4  1          0.8               0.50       100      0.7433300
##   0.4  1          0.8               0.50       150      0.7384965
##   0.4  1          0.8               0.75        50      0.7589632
##   0.4  1          0.8               0.75       100      0.7478994
##   0.4  1          0.8               0.75       150      0.7425844
##   0.4  1          0.8               1.00        50      0.7602074
##   0.4  1          0.8               1.00       100      0.7526321
##   0.4  1          0.8               1.00       150      0.7477816
##   0.4  2          0.6               0.50        50      0.7407292
##   0.4  2          0.6               0.50       100      0.7296555
##   0.4  2          0.6               0.50       150      0.7231783
##   0.4  2          0.6               0.75        50      0.7434259
##   0.4  2          0.6               0.75       100      0.7339772
##   0.4  2          0.6               0.75       150      0.7284584
##   0.4  2          0.6               1.00        50      0.7442404
##   0.4  2          0.6               1.00       100      0.7355150
##   0.4  2          0.6               1.00       150      0.7301150
##   0.4  2          0.8               0.50        50      0.7334168
##   0.4  2          0.8               0.50       100      0.7255845
##   0.4  2          0.8               0.50       150      0.7147372
##   0.4  2          0.8               0.75        50      0.7389953
##   0.4  2          0.8               0.75       100      0.7303682
##   0.4  2          0.8               0.75       150      0.7268248
##   0.4  2          0.8               1.00        50      0.7442663
##   0.4  2          0.8               1.00       100      0.7342825
##   0.4  2          0.8               1.00       150      0.7287683
##   0.4  3          0.6               0.50        50      0.7359378
##   0.4  3          0.6               0.50       100      0.7255934
##   0.4  3          0.6               0.50       150      0.7280381
##   0.4  3          0.6               0.75        50      0.7385790
##   0.4  3          0.6               0.75       100      0.7274135
##   0.4  3          0.6               0.75       150      0.7209702
##   0.4  3          0.6               1.00        50      0.7383441
##   0.4  3          0.6               1.00       100      0.7288983
##   0.4  3          0.6               1.00       150      0.7260075
##   0.4  3          0.8               0.50        50      0.7316068
##   0.4  3          0.8               0.50       100      0.7225762
##   0.4  3          0.8               0.50       150      0.7234279
##   0.4  3          0.8               0.75        50      0.7327961
##   0.4  3          0.8               0.75       100      0.7265637
##   0.4  3          0.8               0.75       150      0.7189373
##   0.4  3          0.8               1.00        50      0.7353974
##   0.4  3          0.8               1.00       100      0.7250317
##   0.4  3          0.8               1.00       150      0.7230962
##   Kappa    
##   0.4462596
##   0.4429630
##   0.4200132
##   0.4658151
##   0.4604554
##   0.4484303
##   0.4592086
##   0.4581448
##   0.4411794
##   0.4564908
##   0.4333714
##   0.4326394
##   0.4617991
##   0.4408186
##   0.4317274
##   0.4598021
##   0.4565921
##   0.4482462
##   0.4303652
##   0.4084279
##   0.3906327
##   0.4429770
##   0.4263209
##   0.4031319
##   0.4407674
##   0.4085965
##   0.4036242
##   0.4374084
##   0.4046743
##   0.3930692
##   0.4319423
##   0.4125149
##   0.4007124
##   0.4442622
##   0.4265787
##   0.4107550
##   0.4324777
##   0.4065217
##   0.3980458
##   0.4093924
##   0.3827307
##   0.3791252
##   0.4174869
##   0.4064532
##   0.3981495
##   0.4083027
##   0.3920819
##   0.3903599
##   0.4169325
##   0.3953971
##   0.3905689
##   0.4316228
##   0.4194146
##   0.4072687
##   0.4598474
##   0.4423625
##   0.4186239
##   0.4571191
##   0.4381275
##   0.4291049
##   0.4676431
##   0.4505231
##   0.4348747
##   0.4497447
##   0.4299946
##   0.4199297
##   0.4587197
##   0.4376588
##   0.4291174
##   0.4594020
##   0.4467976
##   0.4373434
##   0.4236511
##   0.4031415
##   0.3887923
##   0.4296547
##   0.4117075
##   0.3993140
##   0.4293416
##   0.4146539
##   0.4047353
##   0.4085791
##   0.3918990
##   0.3670191
##   0.4200514
##   0.4046337
##   0.3974451
##   0.4300328
##   0.4111720
##   0.4001600
##   0.4127821
##   0.3919866
##   0.3963679
##   0.4192566
##   0.3950423
##   0.3808712
##   0.4182499
##   0.3987796
##   0.3927267
##   0.4073487
##   0.3872865
##   0.3889411
##   0.4100441
##   0.3944626
##   0.3777648
##   0.4144144
##   0.3909891
##   0.3878530
## 
## Tuning parameter 'gamma' was held constant at a value of 0
## 
## Tuning parameter 'min_child_weight' was held constant at a value of 1
## Accuracy was used to select the optimal model using the largest value.
## The final values used for the model were nrounds = 50, max_depth = 1,
##  eta = 0.4, gamma = 0, colsample_bytree = 0.6, min_child_weight = 1
##  and subsample = 1.

Check the Importance of Attributes

varimp_rf <- varImp(model_rf)
varimp_rf
## rf variable importance
## 
##           Overall
## glucose  100.0000
## mass      51.8062
## age       40.3086
## pedigree  28.6711
## pregnant  11.7756
## pressure   9.9492
## triceps    0.4362
## insulin    0.0000

Visualize the Attributes’ Importance

plot(varimp_rf, main = "Variable Importance with Random Forest")

Compare the Performance of Multiple Algorithms

model_comparison <- resamples(list(RF = model_rf, BAG = model_bag, KNN = model_knn,
                                   gbm = model_gbm, xgb = model_xgb))
summary(model_comparison)
## 
## Call:
## summary.resamples(object = model_comparison)
## 
## Models: RF, BAG, KNN, gbm, xgb 
## Number of resamples: 25 
## 
## Accuracy 
##          Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
## RF  0.7247191 0.7487179 0.7653061 0.7624434 0.7777778 0.8088235    0
## BAG 0.6945813 0.7258065 0.7411168 0.7369069 0.7512195 0.7817259    0
## KNN 0.6153846 0.6887755 0.7168950 0.7121062 0.7433155 0.7817259    0
## gbm 0.6787565 0.7389163 0.7606383 0.7581561 0.7794872 0.8031088    0
## xgb 0.6881188 0.7512438 0.7692308 0.7638681 0.7817259 0.8115942    0
## 
## Kappa 
##          Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
## RF  0.3724277 0.4234770 0.4718310 0.4640791 0.5022061 0.5508130    0
## BAG 0.3077769 0.3760277 0.4247151 0.4135043 0.4444444 0.5044170    0
## KNN 0.1355872 0.2960546 0.3634009 0.3449934 0.4008293 0.4835935    0
## gbm 0.3096804 0.4007092 0.4512535 0.4428866 0.4813896 0.5407064    0
## xgb 0.2940197 0.4287147 0.4817761 0.4676431 0.5098131 0.5910956    0

Graphically Compare Performance

scales <- list(x = list(relation = "free"),
               y = list(relation = "free"))

bwplot(model_comparison, scales = scales)

Load in Necessary Python packages

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, ShuffleSplit, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn import metrics
from scipy.spatial.distance import cdist
from sklearn.pipeline import make_pipeline
import matplotlib.pyplot as plt

Data Preparation and KNN modeling

diabetes = pd.read_csv("https://datahub.io/machine-learning/diabetes/r/diabetes.csv")
X = diabetes.drop('class', axis=1)
y = diabetes['class']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=16)
classifier = KNeighborsClassifier()
classifier.fit(X_train, y_train)
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=5, p=2,
##                      weights='uniform')
y_pred = classifier.predict(X_test)
print(f"Accuracy: {round(metrics.accuracy_score(y_test, y_pred)*100, 2)}%")
## Accuracy: 72.29%
df_confusion = pd.crosstab(y_test, y_pred)
df_confusion
## col_0            tested_negative  tested_positive
## class                                            
## tested_negative              124               23
## tested_positive               41               43

Preprocess Data: Standardization/Normalization

import warnings
warnings.filterwarnings("ignore")

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
## StandardScaler(copy=True, with_mean=True, with_std=True)
X_train_std = scaler.transform(X_train)
X_test_std = scaler.transform(X_test)
pd.DataFrame(X_train_std).mean()
## 0   -2.439596e-17
## 1    1.868979e-16
## 2    7.153392e-17
## 3    5.933594e-17
## 4    1.281822e-17
## 5   -2.456136e-16
## 6    8.042398e-17
## 7    4.507051e-17
## dtype: float64
X_train_std.std()
## 1.0
X_train.mean()
## preg      3.780261
## plas    120.109870
## pres     68.823091
## skin     21.016760
## insu     81.446927
## mass     32.067784
## pedi      0.475715
## age      32.957169
## dtype: float64
X_train.std()
## preg      3.309324
## plas     32.972357
## pres     19.563244
## skin     16.123755
## insu    115.457871
## mass      8.165811
## pedi      0.332525
## age      11.459534
## dtype: float64

KNN Performance Evaluation

classifier_std = KNeighborsClassifier()
classifier_std.fit(X_train_std, y_train)
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=5, p=2,
##                      weights='uniform')
y_pred_std = classifier_std.predict(X_test_std)
pd.Series(y_pred_std).value_counts()
## tested_negative    165
## tested_positive     66
## dtype: int64
value, count = np.unique(y_pred_std, return_counts=True)
pd.DataFrame({"value": value, "count": count})
##              value  count
## 0  tested_negative    165
## 1  tested_positive     66
print(f"Accuracy: {round(metrics.accuracy_score(y_test, y_pred_std)*100, 2)}%")
## Accuracy: 74.89%

Use Pipeline to Streamline the Analysis

knn_pipe = make_pipeline(StandardScaler(), KNeighborsClassifier())
knn_pipe.fit(X_train, y_train)
## Pipeline(memory=None,
##          steps=[('standardscaler',
##                  StandardScaler(copy=True, with_mean=True, with_std=True)),
##                 ('kneighborsclassifier',
##                  KNeighborsClassifier(algorithm='auto', leaf_size=30,
##                                       metric='minkowski', metric_params=None,
##                                       n_jobs=None, n_neighbors=5, p=2,
##                                       weights='uniform'))],
##          verbose=False)
pipe_pred = knn_pipe.predict(X_test)
pd.Series(pipe_pred).value_counts()
## tested_negative    165
## tested_positive     66
## dtype: int64

Get Repeated Hold Out Accurary of Model

cv = ShuffleSplit(n_splits=100, test_size=0.3, random_state=16)
from sklearn.model_selection import KFold
cv = KFold(n_splits=10, shuffle=True, random_state=16)
cross_val_score(knn_pipe, X_train, y_train, cv=cv).mean()
## 0.7022361984626136

Sensitivity Analysis

from matplotlib.legend_handler import HandlerLine2D
neighbors = list(range(1, 30))
train_results = []
test_results = []
for n in neighbors:
  model = KNeighborsClassifier(n_neighbors=n)
  model.fit(X_train_std, y_train)
  train_pred = model.predict(X_train_std)
  acc = cross_val_score(model, X_train_std, y_train, cv=cv).mean()*100
  train_results.append(acc)
  y_pred = model.predict(X_test_std)
  acc_test = round(metrics.accuracy_score(y_test, y_pred)*100, 2)
  test_results.append(acc_test)
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=1, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=2, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=3, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=4, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=5, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=6, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=7, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=8, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=9, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=10, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=11, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=12, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=13, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=14, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=15, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=16, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=17, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=18, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=19, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=20, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=21, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=22, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=23, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=24, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=25, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=26, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=27, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=28, p=2,
##                      weights='uniform')
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
##                      metric_params=None, n_jobs=None, n_neighbors=29, p=2,
##                      weights='uniform')
line1, = plt.plot(neighbors, train_results, 'b', label="Train Accuracy")
line2, = plt.plot(neighbors, test_results, 'r', label="Test Accuracy")
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("Accuracy")
plt.xlabel("n_neighbors")
plt.show()

Ensemble Learning - Bagging

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

cart = DecisionTreeClassifier()
num_trees = 100
model = BaggingClassifier(base_estimator=cart, n_estimators=num_trees, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy: {round(results.mean()*100, 2)}%")
## Accuracy: 71.73%

Random Forest

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=num_trees, max_features=5, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy: {round(results.mean()*100, 2)}%")
## Accuracy: 73.59%
model
## RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
##                        max_depth=None, max_features=5, max_leaf_nodes=None,
##                        min_impurity_decrease=0.0, min_impurity_split=None,
##                        min_samples_leaf=1, min_samples_split=2,
##                        min_weight_fraction_leaf=0.0, n_estimators=100,
##                        n_jobs=None, oob_score=False, random_state=16, verbose=0,
##                        warm_start=False)

Feature Importance

model.fit(X_train_std, y_train)
## RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
##                        max_depth=None, max_features=5, max_leaf_nodes=None,
##                        min_impurity_decrease=0.0, min_impurity_split=None,
##                        min_samples_leaf=1, min_samples_split=2,
##                        min_weight_fraction_leaf=0.0, n_estimators=100,
##                        n_jobs=None, oob_score=False, random_state=16, verbose=0,
##                        warm_start=False)
feature_imp = pd.DataFrame(model.feature_importances_, index=X_train.columns,
columns=['importance']).sort_values('importance', ascending=False)
feature_imp
##       importance
## plas    0.299555
## mass    0.192756
## pedi    0.133711
## age     0.105525
## pres    0.083997
## preg    0.067676
## insu    0.059420
## skin    0.057359

AdaBoosting

from sklearn.ensemble import AdaBoostClassifier
model = AdaBoostClassifier(n_estimators=num_trees, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy: {round(results.mean()*100, 2)}%")
## Accuracy: 72.08%
model
## AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1.0,
##                    n_estimators=100, random_state=16)

GBM

from sklearn.ensemble import GradientBoostingClassifier as gbm
model = gbm(n_estimators=num_trees, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy for GBM: {round(results.mean()*100, 2)}%")
## Accuracy for GBM: 72.47%
model
## GradientBoostingClassifier(criterion='friedman_mse', init=None,
##                            learning_rate=0.1, loss='deviance', max_depth=3,
##                            max_features=None, max_leaf_nodes=None,
##                            min_impurity_decrease=0.0, min_impurity_split=None,
##                            min_samples_leaf=1, min_samples_split=2,
##                            min_weight_fraction_leaf=0.0, n_estimators=100,
##                            n_iter_no_change=None, presort='auto',
##                            random_state=16, subsample=1.0, tol=0.0001,
##                            validation_fraction=0.1, verbose=0,
##                            warm_start=False)

Model Tuning

from sklearn.model_selection import GridSearchCV
param_grid = {'learning_rate': np.arange(0.02, 0.1, 0.02),
              'n_estimators': range(60, 160, 50),
              'max_depth': range(2, 5)}
clf = GridSearchCV(gbm(), param_grid)
clf.fit(X_train_std, y_train)
## GridSearchCV(cv='warn', error_score='raise-deprecating',
##              estimator=GradientBoostingClassifier(criterion='friedman_mse',
##                                                   init=None, learning_rate=0.1,
##                                                   loss='deviance', max_depth=3,
##                                                   max_features=None,
##                                                   max_leaf_nodes=None,
##                                                   min_impurity_decrease=0.0,
##                                                   min_impurity_split=None,
##                                                   min_samples_leaf=1,
##                                                   min_samples_split=2,
##                                                   min_weight_fraction_leaf=0.0,
##                                                   n_estimators=100,
##                                                   n_iter_no_change=None,
##                                                   presort='auto',
##                                                   random_state=None,
##                                                   subsample=1.0, tol=0.0001,
##                                                   validation_fraction=0.1,
##                                                   verbose=0, warm_start=False),
##              iid='warn', n_jobs=None,
##              param_grid={'learning_rate': array([0.02, 0.04, 0.06, 0.08]),
##                          'max_depth': range(2, 5),
##                          'n_estimators': range(60, 160, 50)},
##              pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
##              scoring=None, verbose=0)

Model Tuning (2)

print(f"Accuracy for best GBM: {round(clf.best_score_*100, 2)}%")
## Accuracy for best GBM: 75.23%
for key, val in clf.best_params_.items():
  print(f"Best hyperparameter is {key}: {val}")
## Best hyperparameter is learning_rate: 0.02
## Best hyperparameter is max_depth: 2
## Best hyperparameter is n_estimators: 110